"Illegal temporal ID in RTP/HEVC packet" error from ffmpeg when publishing HEVC stream with SRT and reading with RTSP

Which version are you using?
v1.9.3
Which operating system are you using?
Linux amd64 standard, Windows amd64 standard
Describe how to replicate the issue
Publish an HEVC stream with SRT to the server
ffmpeg -re -f lavfi -i testsrc -force_key_frames "expr:gte(t,n_forced*1)" -c:v libx265 -f mpegts srt://localhost:8890/?streamid=publish:test_hevc
then read it with RTSP
ffmpeg -i rtsp://localhost:8554/test_hevc -c copy -f null nul
You should see the error happening after a second. It seems to occur on every keyframe, which is why I forced keyframe every second. This also seems to only happen with SRT->RTSP, I tested RTSP->RTSP and there's no problem there.
Server logs
No response
Network dump
No response

I have a similar problem. I'm trying to get an additional reduced hls stream.
My mediamtx.xml:
runOnReady: \
ffmpeg -hide_banner -y -fflags nobuffer -i rtsp://localhost/mystream \
-vf scale=w=640:h=360 -tag:v hvc1 -c:v libx265 -c:a copy -preset veryfast -profile:v main -hls_time 1 -hls_list_size 7 -hls_flags delete_segments -start_number 100 -b:v 300k -maxrate 856k -bufsize 1200k -b:a 60k -hls_segment_type fmp4 -hls_segment_filename /var/www/html/stream/%03d.mp4 /var/www/html/stream/360p.m3u8

Mediamtx logs:
2024/11/30 18:29:53 INF MediaMTX v1.9.3
2024/11/30 18:29:53 INF configuration loaded from /mediamtx.yml
2024/11/30 18:29:53 INF [RTSP] listener opened on :8554 (TCP), :8202 (UDP/RTP), :8203 (UDP/RT                                                          CP)
2024/11/30 18:29:53 INF [RTMP] listener opened on :1935
2024/11/30 18:29:53 INF [HLS] listener opened on :8888
2024/11/30 18:29:53 INF [WebRTC] listener opened on :8889 (HTTP), :8189 (ICE/UDP)
2024/11/30 18:29:53 INF [SRT] listener opened on :8890 (UDP)
2024/11/30 18:30:00 INF [RTMP] [conn <IP_ADDRESS>:12343] opened
2024/11/30 18:30:01 INF [path mystream] runOnReady command started
2024/11/30 18:30:01 INF [RTMP] [conn <IP_ADDRESS>:12343] is publishing to path 'mystream', 2                                                           tracks (H265, MPEG-4 Audio)
2024/11/30 18:30:01 INF [HLS] [muxer mystream] created automatically
2024/11/30 18:30:01 INF [HLS] [muxer mystream] is converting into HLS, 2 tracks (H265, MPEG-4                                                           Audio)
2024/11/30 18:30:01 INF [RTSP] [conn <IP_ADDRESS>:38538] opened
2024/11/30 18:30:01 INF [RTSP] [session 9dc0a882] created by <IP_ADDRESS>:38538
2024/11/30 18:30:01 INF [RTSP] [session 9dc0a882] is reading from path 'mystream', with UDP,                                                           2 tracks (H265, MPEG-4 Audio)
[rtsp @ 0x7fa392adb600] Illegal temporal ID in RTP/HEVC packet
    Last message repeated 7 times
Input #0, rtsp, from 'rtsp://<IP_ADDRESS>:8554/mystream':
  Metadata:
    title           :
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: hevc (Main), yuv420p(tv, bt709), 1280x720 [SAR 1:1 DAR 16:9], 60 fps, 6                                                          0 tbr, 90k tbn
  Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> hevc (libx265))
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[rtsp @ 0x7fa392adb600] Illegal temporal ID in RTP/HEVC packet
[hevc @ 0x7fa38d8a6940] Could not find ref with POC 34
[hevc @ 0x7fa38d8a6940] Could not find ref with POC 33
[hevc @ 0x7fa38d8a6940] Could not find ref with POC 32
[hevc @ 0x7fa38d8a6940] Could not find ref with POC 31
[rtsp @ 0x7fa392adb600] Illegal temporal ID in RTP/HEVC packet
[hevc @ 0x7fa38d590bc0] Could not find ref with POC 36
[hevc @ 0x7fa38d590bc0] Could not find ref with POC 34
[hevc @ 0x7fa38d590bc0] Could not find ref with POC 33
[hevc @ 0x7fa38d56c880] Could not find ref with POC 38
[hevc @ 0x7fa38d56c880] Could not find ref with POC 36
[hevc @ 0x7fa38d4fc8c0] Could not find ref with POC 38
[hevc @ 0x7fa38d4fc8c0] Could not find ref with POC 36
[rtsp @ 0x7fa392adb600] Illegal temporal ID in RTP/HEVC packet
[hevc @ 0x7fa38d486880] Could not find ref with POC 38
[rtsp @ 0x7fa392adb600] Illegal temporal ID in RTP/HEVC packet
[hevc @ 0x7fa38d457880] Could not find ref with POC 42
[hevc @ 0x7fa38d4488c0] Could not find ref with POC 44
[hevc @ 0x7fa38d4488c0] Could not find ref with POC 42
[rtsp @ 0x7fa392adb600] Illegal temporal ID in RTP/HEVC packet
[hevc @ 0x7fa38d3648c0] Could not find ref with POC 44
[hevc @ 0x7fa38d3648c0] Could not find ref with POC 42
[hevc @ 0x7fa38d35c900] Could not find ref with POC 47
[hevc @ 0x7fa38d35c900] Could not find ref with POC 44
[rtsp @ 0x7fa392adb600] Illegal temporal ID in RTP/HEVC packet
x265 [info]: HEVC encoder version 3.5+1-f0c1022b6
x265 [info]: build info [Linux][GCC 13.2.1][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: Main profile, Level-3 (Main tier)
x265 [info]: Thread pool created using 8 threads
x265 [info]: Slices                              : 1
x265 [info]: frame threads / pool features       : 3 / wpp(6 rows)
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
x265 [warning]: Source height < 720p; disabling lookahead-slices
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge         : hex / 57 / 1 / 2
x265 [info]: Keyframe min / max / scenecut / bias  : 25 / 250 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt        : 15 / 4 / 0
x265 [info]: b-pyramid / weightp / weightb       : 1 / 1 / 0
x265 [info]: References / ref-limit  cu / depth  : 2 / on / on
x265 [info]: AQ: mode / str / qg-size / cu-tree  : 2 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress            : ABR-300 kbps / 0.60
x265 [info]: VBV/HRD buffer / max-rate / init    : 1200 / 856 / 0.750
x265 [info]: tools: rd=2 psy-rd=2.00 early-skip rskip mode=1 signhide tmvp
x265 [info]: tools: fast-intra strong-intra-smoothing deblock sao
[hls @ 0x7fa38d4a80c0] Opening '/var/www/html/stream/init.mp4' for writing
Output #0, hls, to '/var/www/html/stream/360p.m3u8':
  Metadata:
    title           :
    encoder         : Lavf60.16.100
  Stream #0:0: Video: hevc (hvc1 / 0x31637668), yuv420p(tv, bt709, progressive), 640x360 [SAR                                                           1:1 DAR 16:9], q=2-31, 300 kb/s, 60 fps, 15360 tbn
    Metadata:
      encoder         : Lavc60.31.102 libx265
    Side data:
      cpb: bitrate max/min/avg: 856000/0/300000 buffer size: 1200000 vbv_delay: N/A
  Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp, 60 kb/s
frame=    6 fps=0.0 q=46.0 size=N/A time=00:00:00.72 bitrate=N/A dup=30 drop=0 speed= 3.8x                                                             [hevc @ 0x7fa38d8a6940] Could not find ref with POC 47
[hevc @ 0x7fa38d590bc0] Could not find ref with POC 50
[hevc @ 0x7fa38d590bc0] Could not find ref with POC 47
[hevc @ 0x7fa38d56c880] Could not find ref with POC 50
[rtsp @ 0x7fa392adb600] Illegal temporal ID in RTP/HEVC packet
[hevc @ 0x7fa38d4fc8c0] Could not find ref with POC 50
[hevc @ 0x7fa38d486880] Could not find ref with POC 50
[rtsp @ 0x7fa392adb600] Illegal temporal ID in RTP/HEVC packet
[hevc @ 0x7fa38d4488c0] Could not find ref with POC 56
[rtsp @ 0x7fa392adb600] Illegal temporal ID in RTP/HEVC packet
[hevc @ 0x7fa38d3648c0] Could not find ref with POC 56
[hevc @ 0x7fa38d35c900] Could not find ref with POC 56
[hevc @ 0x7fa38d8a6940] Could not find ref with POC 60
[rtsp @ 0x7fa392adb600] Illegal temporal ID in RTP/HEVC packet
[hevc @ 0x7fa38d590bc0] Could not find ref with POC 62
[hevc @ 0x7fa38d590bc0] Could not find ref with POC 60
[hevc @ 0x7fa38d56c880] Could not find ref with POC 62
[hevc @ 0x7fa38d56c880] Could not find ref with POC 60
....

hls player output:
![Снимок экрана 2024-11-30 214509](https://github.com/user-attachments/assets/6baf34b9-b113-4c48-85f7-dd6a39a301d8)


Doesn't work \SQL formatter

Shows just empty window

It duplicates #105 .

Hi,
DevToysMac is now archived and deprecated. We just released DevToys 2.0 for Windows, Mac and Linux. This issue may or may have not been addressed in the new version. If the issue persist, we encourage to open a new issue on https://github.com/DevToys-app/DevToys or https://github.com/DevToys-app/DevToys.Tools
See more information about DevToys 2.0 here: https://devtoys.app/blog/announcing-devtoys-2.0-preview


Write a Ruby program to find largest number in an array

Description
Write a Ruby program to find largest number in an array
Input  : [1, 2, 3, 4, 5]
Output : 5

How to contribute

Save the solution in program/FindLargestNumberInAnArray.rb file
Add FindLargestNumberInAnArray.rb file in find-largest-number-in-an-array folder

!assign


javolution-serializer: Provide more efficient serialization for StringBuffer and StringBuilder

From martin.grotzke on March 27, 2010 00:36:21
javolution serializer right now does not have a custom xml format for
StringBuffer and StringBuilder and therefore serializes them using
reflection. This produces verbose output, as both hold a Character[]
internally.
Both should be serialized just using the string representation.
Original issue: http://code.google.com/p/memcached-session-manager/issues/detail?id=57

From martin.grotzke on March 26, 2010 16:44:23
Fixed.
Status: Fixed
Labels: -Type-Defect Type-Enhancement Milestone-1.3.0

From martin.grotzke on April 05, 2010 15:15:51
(No comment was entered for this change.)
Labels: -Milestone-1.3.0 Milestone-1.3


Transfer list component

Notes for Reviewers
This PR fixes #387
Signed commits

[x] Yes, I signed my commits.

Do we have a theme reference?

Right Arrow icon PR merged.


Can't enable motion from flow

Describe the bug
I have a flow that disables the motion sensor for 10 minutes, then enables it again. I'm migrating from Hue Magic. When I run the flow, the motion disables but after the timeout, it doesn't come back on.
To Reproduce
Steps to reproduce the behavior:
[{"id":"c13967da.3c8eb","type":"api-call-service","z":"95e7528c.dd36c","name":"Hallway Lights On","server":"7741cb1d.3b20c4","version":5,"debugenabled":false,"domain":"light","service":"turn_on","areaId":[],"deviceId":[],"entityId":["light.hallway_lamp_1_2","light.hallway_lamp_2_2"],"data":"{\"brightness_pct\":\"100\"}","dataType":"json","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","output_location":"","output_location_type":"none","x":610,"y":2160,"wires":[[]]},{"id":"56c13fe5.bdb578","type":"server-events","z":"95e7528c.dd36c","name":"","server":"7741cb1d.3b20c4","version":1,"event_type":"call_service","exposeToHomeAssistant":false,"haConfig":[{"property":"name","value":""},{"property":"icon","value":""}],"waitForRunning":true,"outputProperties":[{"property":"payload","propertyType":"msg","value":"","valueType":"eventData"},{"property":"topic","propertyType":"msg","value":"$outputData(\"eventData\").event_type","valueType":"jsonata"},{"property":"event_type","propertyType":"msg","value":"$outputData(\"eventData\").event_type","valueType":"jsonata"}],"x":110,"y":2260,"wires":[["c161c505.f81688","1ea28cc1.795d83","cf5cb79e.773fe"]]},{"id":"c161c505.f81688","type":"switch","z":"95e7528c.dd36c","name":"Cat Litter Time","property":"payload.event.service_data.entity_id","propertyType":"msg","rules":[{"t":"eq","v":"script.cat_litter_time","vt":"str"}],"checkall":"true","repair":false,"outputs":1,"x":320,"y":2240,"wires":[["c13967da.3c8eb"]]},{"id":"1ea28cc1.795d83","type":"debug","z":"95e7528c.dd36c","name":"cat","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":330,"y":2400,"wires":[]},{"id":"cf5cb79e.773fe","type":"switch","z":"95e7528c.dd36c","name":"Cat Litter Time","property":"payload.event.service","propertyType":"msg","rules":[{"t":"eq","v":"cat_litter_time","vt":"str"}],"checkall":"true","repair":false,"outputs":1,"x":320,"y":2280,"wires":[["c13967da.3c8eb","8a25c2702f5bc1bf","fbe2cfcb2f23f0ce"]]},{"id":"31f859355b193fa6","type":"inject","z":"95e7528c.dd36c","name":"","props":[{"p":"payload.event.service","v":"cat_litter_time","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":90,"y":2340,"wires":[["cf5cb79e.773fe"]]},{"id":"8a25c2702f5bc1bf","type":"trigger","z":"95e7528c.dd36c","name":"","op1":"","op2":"","op1type":"nul","op2type":"date","duration":"10","extend":true,"overrideDelay":false,"units":"s","reset":"STOP","bytopic":"all","topic":"topic","outputs":1,"x":590,"y":2220,"wires":[["6e02db78.1ba654","b4fc6368397392df","662c1aa1b617feae"]]},{"id":"662c1aa1b617feae","type":"debug","z":"95e7528c.dd36c","name":"cat","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":950,"y":2280,"wires":[]},{"id":"6e02db78.1ba654","type":"api-call-service","z":"95e7528c.dd36c","name":"Hallways Lights Off","server":"7741cb1d.3b20c4","version":5,"debugenabled":false,"domain":"light","service":"turn_off","areaId":[],"deviceId":[],"entityId":["light.hallway_lamp_1_2","light.hallway_lamp_2_2"],"data":"","dataType":"json","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","output_location":"","output_location_type":"none","x":910,"y":2160,"wires":[[]]},{"id":"b144094da62d3035","type":"inject","z":"95e7528c.dd36c","name":"","props":[{"p":"payload.event.service","v":"cat_litter_time","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":710,"y":2360,"wires":[["6e02db78.1ba654"]]},{"id":"fbe2cfcb2f23f0ce","type":"link out","z":"95e7528c.dd36c","name":"link out 1","mode":"link","links":["c3e862fe.aaf66"],"x":645,"y":2280,"wires":[]},{"id":"b4fc6368397392df","type":"link out","z":"95e7528c.dd36c","name":"link out 2","mode":"link","links":["b1aa549b.424a38"],"x":835,"y":2220,"wires":[]},{"id":"7741cb1d.3b20c4","type":"server","name":"Home Assistant","version":4,"addon":true,"rejectUnauthorizedCerts":true,"ha_boolean":"y|yes|true|on|home|open","connectionDelay":true,"cacheJson":true,"heartbeat":false,"heartbeatInterval":"30","areaSelector":"friendlyName","deviceSelector":"friendlyName","entitySelector":"friendlyName","statusSeparator":"at: ","statusYear":"hidden","statusMonth":"short","statusDay":"numeric","statusHourCycle":"h23","statusTimeFormat":"h:m"}]
Expected behavior
I expect the motion sensor to be enabled

Hey,
what is this? You are using nodes that are not part of my flow example.
Use my example,  deactivate the sensor, use a delay node for the time you need and activate it again.
There is a description in the labels of my flow how sensor enable and disable api calls work.
I can’t help with your setup.

There are two lights which aren't in Hue, I use Home Assistant.

Is your wish that the lower cat litter time disable - and the top one enable the sensor

And http request for sensors looks like that:

one change node contains the data for enabling it, the other one for disable it.
both go into the header and the http request. You need to add your discovered data in all nodes. Otherwise it won’t work.

If I manually click the timestamp nodes here, the sensor enables and disables as I expect. IT seems to be and issue with the links between my flow and your flow. I've probably done something wrong.

When it works with the inject nodes everything is fine. You probably linked it not correct.
i am closing the issue then.

Enabled

Disabled

@Puntoboy check if your link out are correctly connected to the link in nodes.
Gives them a name as recommended in the readme. Otherwise it’s too confusing

The link node is named.

Where comes this timeout from?
You can’t make too much api calls at once. Please read the readme. You are the first with this problem. It seems to be in your flow.
Debug everything and try to find the error

As I test, I'm linking directly HTTP request. Same issue. This is my only API call at the moment as I'm just setting it up.

As I test, I'm linking directly HTTP request. Same issue. This is my only API call at the moment as I'm just setting it up.

I'm not saying it's a bug with this setup, I've likely done something wrong, but I can't see what it is.

I can see it now. Use a 10 second delay node instead of the trigger node.

Still seeing the JSON parse error.

Neither the disable or enable are working consistently.

Ok, debug the data before it reaches the change nodes. You probably need to delete all extra data that comes from your home assistant node. I am pretty sure now that it mess up the payload.
You can try it. Remove the home assistant node, and inject a Boolean true there at the position. If it works you msg is messed up by that node

@Puntoboy what is events call service?

That listened to events from Home Assistant. The switch node after it, filters for a specific event, in this case when a specific script is run in HA.

@Puntoboy ok… what event. Specific, maybe we find a workaround

@Puntoboy no, the home assistant node. What is the script for?
please debug (full msg object) what comes out of the home assistant node


2 dependencies should be updated

Hey @greysteil
I manually bumped my repo but I got no PR.
locally I got this:

Can you please check this?

I added my composer.json/.lock as a gist here:
https://gist.github.com/OskarStark/715cd909576b5ce4a2ee2971ee196f53

Thanks @OskarStark. Can you let me know the name of your repo, so I can take a look at Dependabot's logs? If you don't want to put it here then you can ping me on<EMAIL_ADDRESS>

its called OskarStark/acd

OK, I can replicate. Digging into this now. Thanks for the report!

Thank you for the great support 👍

Fixed in https://github.com/dependabot/dependabot-core/commit/77406d5d220ce7c61c7d68d1bacb8b8fadf59277, which I'm deploying now. Thanks again for the heads up!

Works like a charme, I already got the first PR 😄

FYI, it looks like the symfony/phpunit-bridge issue is actually a separate one, which I'm fixing now.

And that issue is fixed now too. 🎉


Use organization secret NPM_ORG_TOKEN

Please read https://github.com/maplibre/maplibre-gl-js/pull/231. @nyurik please share the MapLibre organization secret NPM_ORG_TOKEN with this repo.

Thanks Harel :)
This should only be merge once NPM_ORG_TOKEN has been shared with this repo.


fix: auto-scroll height issue

Description

fix issue by passing contentInset's bottom number

change additionalOffset as (this.state.keyboardSpace - extraHeight) in function scrollResponderScrollNativeHandleToKeyboard()

Explanation

additionalOffset in scrollResponderScrollNativeHandleToKeyboard means scroll view's bottom "contentInset"

https://github.com/facebook/react-native/blob/4cc3aa851c181032e8ea441f0cea6459c018c1d8/Libraries/Components/ScrollView/ScrollView.js#L986
/**
   * This method should be used as the callback to onFocus in a TextInputs'
   * parent view. Note that any module using this mixin needs to return
   * the parent view's ref in getScrollViewRef() in order to use this method.
   * @param {number} nodeHandle The TextInput node handle
   * @param {number} additionalOffset The scroll view's bottom "contentInset".
   *        Default is 0.
   * @param {bool} preventNegativeScrolling Whether to allow pulling the content
   *        down to make it meet the keyboard's top. Default is false.
   */
  scrollResponderScrollNativeHandleToKeyboard: <T>(
    nodeHandle: number | React.ElementRef<HostComponent<T>>,
    additionalOffset?: number,
    preventNegativeScrollOffset?: boolean,
  ) => void = <T>(){}

In react-native-keyboard-aware-scroll-view's KeyboardAwareHOC.js, contentInset's bottom is this.state.keyboardSpace

<ScrollableComponent
  {...refProps}
  keyboardDismissMode='interactive'
  contentInset={{ bottom: this.state.keyboardSpace }}

that's why we should pass (this.state.keyboardSpace - extraHeight) to scrollResponderScrollNativeHandleToKeyboard rather than extraHeight

https://github.com/APSL/react-native-keyboard-aware-scroll-view/blob/6cbf7b861856467892cef620adf9af46c4603fa8/lib/KeyboardAwareHOC.js#L307
const additionalOffset = this.state.keyboardSpace - extraHeight
  responder &&
    responder.scrollResponderScrollNativeHandleToKeyboard(
      reactNode,
      additionalOffset,
      true
    )

Screenshots

After
Before

App

Node version need to be upgraded in CI as per https://github.com/APSL/react-native-keyboard-aware-scroll-view/pull/470


[SPARK-12355][SQL] Implement unhandledFilter interface for Parquet

https://issues.apache.org/jira/browse/SPARK-12355
This is similar with https://github.com/apache/spark/pull/10427.
As discussed here https://github.com/apache/spark/pull/10221, this PR implemented unhandledFilter to remove duplicated Spark-side filtering.
In case of Parquet, the columns referenced in pushed down filters should be given to org.apache.spark.sql.parquet.row.requested_schema whereas general datasources such as JDBC do not require the columns.
However, DataSourceStrategy.pruneFilterProjectRaw() removes the columns only referenced in pushed down filters. Therefore, this PR resolved this problem by manually generating the columns referenced in pushed down filters.

Test build #48396 has started for PR 10502 at commit cf331a4.

Test build #48396 has finished for PR 10502 at commit cf331a4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48396/
Test FAILed.

The test is failed from wrong results from Parquet.
The test result was below:
== Physical Plan ==
Scan ParquetRelation[_1#4] InputPaths: file:/private/var/folders/9j/gf_c342d7d150mwrxvkqnc180000gn/T/spark-2fa8f75a-8ac1-4273-afcd-f271529a24a7, PushedFilters: [GreaterThan(_1,5)]

== Results ==
!== Correct Answer - 4 ==   == Spark Answer - 5 ==
![6]                        [5]
![7]                        [6]
![8]                        [7]
![9]                        [8]
!                           [9]

As can be seen, GreaterThan(_1,5) is pushed down correctly but the results includes 5.
I am looking into this deeper.

I see. UnsafeRowParquetRecordReader at Parquet does not support filter record by record but just block. So, even with = operator produces the same results below.
+---+
| _1|
+---+
|  5|
|  6|
|  7|
|  8|
|  9|
+---+

I think I should disable unhandledFilters when

retest this please

Test build #48403 has started for PR 10502 at commit 8c376af.

Test build #48402 has started for PR 10502 at commit 8c376af.

Test build #48403 has finished for PR 10502 at commit 8c376af.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48403/
Test FAILed.

Merged build finished. Test FAILed.

Test build #48402 has finished for PR 10502 at commit 8c376af.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48402/
Test FAILed.

Test build #48405 has started for PR 10502 at commit 2ad8182.

Test build #48405 has finished for PR 10502 at commit 2ad8182.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48405/
Test PASSed.

Merged build finished. Test PASSed.

cc @yhuai @liancheng

@HyukjinKwon Thank you for the PR? Can you post some benchmarking results (with your testing code)? It will be good to have these numbers to help others understand if it can provide benefit.

@yhuai Sure. I will try!

Benchmark (Removed Spark-side Filter)
Motivation
This PR simplifies the query plans for Parquet files by stripping duplicated Spark-side filtering,
from:
== Physical Plan ==
Filter (a#8 = 2)
  +- Scan ParquetRelation[a#8] InputPaths: file:/private/var/folders/9j/gf_c342d7d150mwrxvkqnc180000gn/T/spark-ef271ec6-95e1-43ae-9b3e-1d4dae6f69c3/part=1, PushedFilters: [EqualTo(a,2)]

to :
== Physical Plan ==
Scan ParquetRelation[a#8] InputPaths: file:/private/var/folders/9j/gf_c342d7d150mwrxvkqnc180000gn/T/spark-ef271ec6-95e1-43ae-9b3e-1d4dae6f69c3/part=1, PushedFilters: [EqualTo(a,2)]

However, in terms of performance, it is unkown if there is benefit. So, this benchmark was performed.
Simply, several quries were executed with duplicated Spark-side filtering and without it.
Environment

Machine: MacBook Pro Retina
CPU: 4
Memory: 8G

Method
In this benchmark,

The filters that Parquet can generate were composed with 9 simple quries. IS NULL, IS NOT NULL, =, !=, <=>, <, >, <= and >= operators were tested here. Each query was executed 5 times and calculated the average times.
The commit right before this PR was used against this PR. Namely, The tests were performed with Spark-side filtering and without Spark-side filtering.
In order to test cleanly, the target Parquet file was uncompressed and in order to enable Parquet filtering row by row, spark.sql.parquet.enableUnsafeRowRecordReader was disabled.

Dataset
Raw Data

TPC-H Lineitem Table created with factor 1 (generate data)
Size : 724.66 MB

Create Target Parquet File

case class for Lineitem table

case class Lineitem(l_orderkey: Int,
                    l_partkey: Int,
                    l_suppkey: Int,
                    l_linenumber: Int,
                    l_quantity: Float,
                    l_extendedprice: Float,
                    l_discount: Float,
                    l_tax: Float,
                    l_returnflag: String,
                    l_linestatus: String,
                    l_shipdate: String,
                    l_commitdate: String,
                    l_receiptdate: String,
                    l_shipinstruct: String,
                    l_shipmode: String,
                    l_comment: String)

Create Parquet file

import sqlContext.implicits._
var conf = new SparkConf()
conf.setAppName("Test").setMaster("local")
conf.set("spark.sql.parquet.compression.codec", "uncompressed")
val sc = new SparkContext(conf)
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
sc.textFile("lineitem.tbl").map(_.split('|')).map { v =>
  Lineitem(
    v(0).trim.toInt,
    v(1).trim.toInt,
    v(2).trim.toInt,
    v(3).trim.toInt,
    v(4).trim.toFloat,
    v(5).trim.toFloat,
    v(6).trim.toFloat,
    v(7).trim.toFloat,
    v(8),
    v(9),
    v(10),
    v(11),
    v(12),
    v(13),
    v(14),
    v(15))
}.toDF()
df.save("lineitem", "parquet")

Parquet file
-rw-r--r--  hyukjinkwon hyukjinkwon 0 B         _SUCCESS
-rw-r--r--  hyukjinkwon hyukjinkwon 1.54 KB     _common_metadata
-rw-r--r--  hyukjinkwon hyukjinkwon 13.72 KB    _metadata
-rw-r--r--  hyukjinkwon hyukjinkwon 52.16 MB    part-r-00000-2aa739b2-6194-47db-9dd5-461905cea976.parquet
-rw-r--r--  hyukjinkwon hyukjinkwon 51.74 MB    part-r-00001-2aa739b2-6194-47db-9dd5-461905cea976.parquet
-rw-r--r--  hyukjinkwon hyukjinkwon 51.97 MB    part-r-00002-2aa739b2-6194-47db-9dd5-461905cea976.parquet
-rw-r--r--  hyukjinkwon hyukjinkwon 51.71 MB    part-r-00003-2aa739b2-6194-47db-9dd5-461905cea976.parquet
-rw-r--r--  hyukjinkwon hyukjinkwon 51.89 MB    part-r-00004-2aa739b2-6194-47db-9dd5-461905cea976.parquet
-rw-r--r--  hyukjinkwon hyukjinkwon 34.25 MB    part-r-00005-2aa739b2-6194-47db-9dd5-461905cea976.parquet

Test Codes

Function to measure time

def time[A](f: => A) = {
  val s = System.nanoTime
  val ret = f
  println("time: "+(System.nanoTime-s)/1e6+"ms")
  ret
}

Configuration and SQLContext

var conf = new SparkConf()
conf.setAppName("Test").setMaster("local")
conf.set("spark.sql.parquet.enableUnsafeRowRecordReader", "false")
val sc = new SparkContext(conf)
val sqlContext = new org.apache.spark.sql.SQLContext(sc)

IS NULL

val source = sqlContext.read.parquet("lineitem")
val df = source.filter("l_orderkey IS NULL").select("l_orderkey")
time(df.collect())

IS NOT NULL

val source = sqlContext.read.parquet("lineitem")
val df = source.filter("l_orderkey IS NOT NULL").select("l_orderkey")
time(df.collect())

=

val source = sqlContext.read.parquet("lineitem")
val df = source.filter("l_orderkey = 1").select("l_orderkey")
time(df.collect())

!=

val source = sqlContext.read.parquet("lineitem")
val df = source.filter("l_orderkey != 1").select("l_orderkey")
time(df.collect())

<=>

val source = sqlContext.read.parquet("lineitem")
val df = source.filter("l_orderkey <=> 1").select("l_orderkey")
time(df.collect())

<

val source = sqlContext.read.parquet("lineitem")
val df = source.filter("l_orderkey < 3000000").select("l_orderkey")
time(df.collect())

>

val source = sqlContext.read.parquet("lineitem")
val df = source.filter("l_orderkey > 3000000").select("l_orderkey")
time(df.collect())

<=

val source = sqlContext.read.parquet("lineitem")
val df = source.filter("l_orderkey <= 3000000").select("l_orderkey")
time(df.collect())

>=

val source = sqlContext.read.parquet("lineitem")
val df = source.filter("l_orderkey >= 3000000").select("l_orderkey")
time(df.collect())

Results

Operator
Without Spark Filtering(ms)
With Spark Filtering(ms)
Time Decreased(%)

IS NULL
645.015
669.038
3.590

IS NOT NULL
8040.593
8394.950
4.221

=
885.764
906.658
2.304

!=
7844.248
8082.113
2.943

<=>
869.402
912.457
4.718

<
4510.599
4625.214
2.478

>
4732.729
4940.915
4.213

<=
4868.453
4918.441
1.016

>=
4751.772
4946.939
3.945

Basically, in a simple view, the difference was below.
The original codes would work as below (With Spark Filtering):
data
  // Parquet-side filtering
  .filter(pushedFilter)
  // Spark-side filtering
  .filter(pushedFilter)

This PR would change this into below (Without Spark Filtering):
data
  // Parquet-side filtering
  .filter(pushedFilter)

Although both the same O(n) time complexity, the former was 2n and the latter was n. So, it seems there is performance benefit.

@yhuai Would you look through this please?

Test build #49115 has started for PR 10502 at commit dfc7506.

Jenkins, retest this please.

Test build #49122 has started for PR 10502 at commit dfc7506.

Test build #49115 has finished for PR 10502 at commit dfc7506.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49115/
Test PASSed.

Test build #49122 has finished for PR 10502 at commit dfc7506.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49122/
Test FAILed.

Merged build finished. Test FAILed.

I will resolve the conflict on Monday.

@yhuai Would you look through this please?

Test build #51075 has started for PR 10502 at commit ad757be.

Test build #51075 has finished for PR 10502 at commit ad757be.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51075/
Test PASSed.

@yhuai Would you look through this please?

Test build #52014 has started for PR 10502 at commit 0e149da.

Test build #52014 has finished for PR 10502 at commit 0e149da.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52014/
Test PASSed.

@HyukjinKwon Thank you for working on it. We have been actively improving the efficiency of code-gen and unsafe parquet reader. For the long term, letting parquet to evaluate filters like a=10 row by row is not as efficient as using the code gen version of Spark filter. We do plan to let unsafe parquet reader to evaluate very cheap predicates like IsNotNull and IsNull for every row. However, before it happens, can we hold off the work on this PR?

@yhuai No problem. Then, please inform me when I am supposed to do something else.

yea will do. Thank you.

Let me close this for now. Please let me know although this is closed. I will reopen this when I start to work on this again.

Hi @yhuai ! Would this be okay if I give a try for this one again maybe?


Nuxt 3 Support

Hey :wave: It looks like this module hasn't been updated for Nuxt 3. In an effort to improve the developer experience of modules, I've updated the module to clarify that it only supports Nuxt 2.
@lihbr Given https://github.com/nuxt-community/ackee-module/issues/15, do you think it's worth archiving this module?

I think it might be worth archiving yes, if you have permissions for this (looks like I don't).
Do we want to put a disclaimer in case anyone wants to take over the maintainership of this module that we're happy to unarchive it? Happy to add that~
Thanks for clarifying this!

I'll hold off on doing anything until the next team meeting and we can discuss what the best path forward is as there are a number of modules in a similar position.


Version 1.15 not backwards compatible with version 1.14 and older

1.15 needs a newer node (>=8.16) version than 1.14 (>=6.0).
This breaks a package depending on this one for me, because I have Node 8.10 (Default on Ubuntu 18.04 LTS) and that package specifies "twig": "1.10.5" as dependency.
Because of this, version 1.15 actually should be a version 2.0.0.

Unfortunately, there were several breaking changes that should have been tagged as v2.0.0. This did not happen though, so there isn't much that can be done at this point. There shouldn't be any more breaking changes before v2.0.0.


Where is the BOOT button on the T-Embed please?

Have one of each of T-Display, the Touch version and the T-Embed versions.  Excellent boards.
However, I am aware there is a TIP listed on the T-Display GitHub page to reset the T-Display and Touch versions by holding down the BOOT button while connecting the USB, then releasing and re uploading the code etc... Unfortunately I appear to have a non working T-Embed device and I want to do the same procedure but can't see where any BOOT button is located on the board, or how I do this boot type reset on the T-Embed device.
Could someone please let me know the similar procedure as to how I get the T-Embed device to reboot using the holding down the boot button method please?
Ta

https://github.com/Xinyuan-LilyGO/T-Embed

Yep, I have seen that page and I see there is a reference to 'Boot' in one of the diagrams towards the bottom of the page, but there is no actual physical boot button that I can see on the board that would allow me to press it whilst connecting the USB cable to start that process!!

As this is the T-display, i just thought a comment in the embed section may gain a response from people who have the T-Embed. Here, most just have the T-display.

Apparently the BOOT button on the T-Embed is the encoder button, but when I try the same process by holding down the encoder button, it still does not do the full re-boot process.  It appears I may have a dead display on my T-Embed sadly.

Hold boot pressed, Press and release reset button, then release the boot button

Hold boot pressed, Press and release reset button, then release the boot button. Don't give up on your display...TTgo/lillygo have poor support. Put me off using these in commercial products. Good to play with though.

So when do I actually do this please...? When in this series of button presses do I connect the USB serial cable...?
And yes, I agree with you 100% - I really do like the boards, but support is sorely lacking.  My sincere thanks goes out to good folk such as you for helping as you do.  Cheers... I owe you a pint :)

Hold boot pressed, Press and release reset button, then release the boot button. Don't give up on your display...TTgo/lillygo have poor support. Put me off using these in commercial products. Good to play with though.

I've tried doing as you suggest but I reckon I received a dead display.  Code appears to compile and upload successfully, the LED's do some stuff, but the display sadly, remains as dead as a Dodo!!!

Did you raise an Issue over at the T-Embed GitHub repository ?
I do not have a T-Embed to test…but a common problem with the LilyGO T-Display S3 is the TFT_eSPI library must be the EXACT one supplied for your model of LilyGO  and not the latest Bodmer and certainly NOT the one supplied by Arduino OR doing a “Arduino Library Update"
Cheers !
Terry Stainton
@.***

On Mar 1, 2023, at 10:06 AM, John @.***> wrote:
Hold boot pressed, Press and release reset button, then release the boot button. Don't give up on your display...TTgo/lillygo have poor support. Put me off using these in commercial products. Good to play with though.
I've tried doing as you suggest but I reckon I received a dead display. Code appears to compile and upload successfully, the LED's do some stuff, but the display sadly, remains as dead as a Dodo!!!
—
Reply to this email directly, view it on GitHub https://github.com/Xinyuan-LilyGO/T-Display-S3/issues/110#issuecomment-1450304957, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATCNKOH7TRF34CMNJIMEXR3WZ5QZHANCNFSM6AAAAAAVLXBMEQ.
You are receiving this because you are subscribed to this thread.

T-Embed  dT-Embed's BOOT button is located in the middle of the rotary encoder. Sorry, this reply is a little late. I hope you have solved this problem

T-Embed dT-Embed's BOOT button is located in the middle of the rotary encoder. Sorry, this reply is a little late. I hope you have solved this problem

Thanks Lewis, I have managed to resolve the issue but it wasn't due to the BOOT button reset process, it was down to the T-Embed display drivers and the TFT_eSPI library - which if used with this board, required a specific TFT_eSPI library available on the T-Embed LilyGo GotHub pages, and NOT the one generally available library from Bodmer.

This warehouse was previously maintained by Micky, and may not be used for some reasons<TFT_ ESPI>, if I have time next, I will use Bodmer's<TFT_ ESPI>test to see if it can be used


Is there a type definition file for ramda for use with ternjs?

Ternjs comes with pre made type definiton files for some commonly used libraries. This is a straightforward json format.
http://ternjs.net/doc/manual.html#typedef

JSON type definitions
To be able to specify the types of things without actually analyzing source code, either because there is no JavaScript source code (as for the built-in types) or because the source code is too big, or because Tern is unable to construct the correct types from the source code, Tern defines a JSON data format for specifying types. A few examples of files in this format can be found in the defs/ directory in the distribution.
A type definition data structure is basically a tree of objects, where the top-level object specifies variables in the global scope, and the nested objects specify properties of object types. Properties prefixed with an exclamation point (!) hold special directives, all other properties refer to variable or property names.
Here is an example:
{
  "!name": "mylibrary",
  "!define": {
    "point": {
      "x": "number",
      "y": "number"
    }
  },
  "MyConstructor": {
    "!type": "fn(arg: string)",
    "staticFunction": "fn() -> bool",
    "prototype": {
      "property": "[number]",
      "clone": "fn() -> +MyConstructor",
      "getPoint": "fn(i: number) -> point"
    }
  },
  "someOtherGlobal": "string"
}

These are the current typedef files that comes with ternjs.. Some widely used libraries such as jquery, react and underscore. I've looked for typedef files made by third parties, but I have not found any others.
Tern is able to automatically analyse most javascript code, but it has problems with ramda, presumably because most functions are higher order ( wrapped in _curry2(), for example).
There's an issue about using ternjs with ramda that has been closed, but not really resolved.
https://github.com/ternjs/tern/issues/570
Would it be possible to compile a ternjs def file for ramda and include it either with ramda or with tern?
If not, how could someone create a def file for themselves? It seems that all the relevant data is already present in the ramda jdocs, but I'm not familiar with the tools needed to work with that.

Would it be possible to compile a ternjs def file for ramda and include it either with ramda or with tern?

I don't know anything about tern, but reading over what you've linked and looking at the typedef files, it does seem quite possible to automatically generate a tern typedef file.
The jsdocs are included in releases in dist/ramda.js, so you could host a repo that depends on ramda, reads that file, and generates something like https://github.com/ternjs/tern/blob/master/defs/underscore.json to either be hosted by them or yourself (not sure what is preferable with tern).

Or if someone wants to do the work, I can easily see this being included in our build, with the file hosted somewhere on http://ramdajs.com.


Explore converting the Travis CI to instead use GitHub actions

@FFY00 has set up GitHub actions to generate Arch Linux packages for SymbiFlow at https://github.com/FFY00/symbiflow-arch-pkgs
This has a couple of really nice aspects;

It generates a list of tasks to run based on which packages have been modified.
It can potentially have a much clearer dependency tree described.

It seems that GitHub actions are the future here. @FFY00 has said he would be interesting to explore using it instead.

Ideally the CI would do the following;
(a) Build the packages and upload to a temporary location
(b) Check that the newly uploaded packages don't break building or running of the dependent packages
(c) Check important test examples (such at https://github.com/timvideos/litex-buildenv , https://github.com/SymbiFlow/symbiflow-examples , https://github.com/SymbiFlow/sv-tests and something around OpenTitan) are not broken by the new packages.
(e) Mark the new packages as ready to be used!

@PiotrZierhoffer - poke?


Cannot invoke graphql subscriptions/websocket APIs when choreo analytics is enabled

Description:
Cannot invoke graphql subscriptions/websocket APIs when choreo analytics is enabled. Getting below error when trying to establish connection.
ERROR - InboundWebsocketSourceHandler Endpoint not found for port : 8099 tenant domain : null

In docs it is advised to access the http url of the EP and accept the cert to resolve this issue. But cannot access the http EP.

Steps to reproduce:

Enable analytics
Create a graphql API and tryout a subscription

Affected Product Version:
Workd fine in 4.1.0 beta pack, broken in latest build (14th March)
Environment details (with versions):

OS:
Client:
Env (Docker/K8s):

Optional Fields
Related Issues:

Suggested Labels:

Suggested Assignees:

This issue was solved after upgrading org.wso2.am.analytics.publisher.client to version 1.0.8.


Disabled tabs are enabled when swipeable is activated

Expected Behavior
When a tab has the class .disabled it shouldn't be accessible
Current Behavior

I used the test code from the documentation:

      Test 1

      Test 2

      Disabled Tab

      Test 4

Test 1

Test 2

Test 3

Test 4

and the disabled class worked fine, except when I set swipeable: true in the options. I still couldn't click to open the tab, but I could swipe into it, like this:

Also got the same problem on Android

Steps to Reproduce (for bugs)
Already described in current behavior
Environment

Version used: both 0.100.2 and 1.0.0-rc.2
Browser Name and version: Google Chrome Version 67.0.3396.87 (Official Build) (64-bit) on desktop, Google Chrome 67.0.3396.87 on Android
Operating System and version (desktop or mobile): Ubuntu 16.04 and Android 8.0.0

If you do, please make sure to branch off v1-dev

Okay, not really sure if I did it right. This is my first it time contributing. But the idea is just to add a condition to prevent disabled tab content from being added to the carousel.
          if(!$(link).parent('li.tab').hasClass('disabled')){             let $currContent = $(M.escapeHash(link.hash));             $currContent.addClass('carousel-item');             $tabsContent = $tabsContent.add($currContent);           }
https://github.com/Dogfalo/materialize/pull/6155/commits/47a58c6b65553790d073f1bbce5af1c8b65b0a08


refactor: Updates Postgres extractor to not use information_schema

Signed-off-by: verdan<EMAIL_ADDRESS>
Summary of Changes
In PostgreSQL information_schema is limited to only tables you have access to. However, the pg_catalog tables are not.
This means that if we want to do a metadata-only integration via a User that has read access to zero tables, at present they'd have to do a materialized view of information_schema.
This PR makes use of pg_catalog instead properly, to make sure the data is ingested for all users, even with no access.
Tests

Documentation

CheckList
Make sure you have checked all steps below to ensure a timely review.

[x] PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"

In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

[x] PR includes a summary of changes.
[x] PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
[x] In case of new functionality, my PR adds documentation that describes how to use it.

All the public functions and the classes in the PR contain docstrings that explain what it does

@verdan could we delete the local branch once it is merged? otherwise the local branch will show up in every repo that forked by others. thx


Missing FPU stack number

Settings:
var formatter = new MasmFormatter();
formatter.Options.DigitSeparator = "";

Input:
{ 0xD8, 0xC9 }
produces:
fmul st,st(1)
expected:
fmul st(0),st(1)

The output is valid masm syntax.
An option to use st(0) could be added since masm also accepts st(0).

That's what I was about to say too. Both is valid but I prefer its showing (0) but it currently does not and i couldn't find a setting for it so it would be nice to have one.


Great first implementation.  We can tweak this a bit more as we need.  But it works pretty well as is.


Mute mypy errors for "check-code" CI

Description
Related to #83, it ensures that "Run typing tests" step does not raises and continues without breaking the validation.
This will simplify the validation of other PRs (at least for code formatting).
The analysis will be performed to be able to review issues and solve it in parallel.
Once #83 will be solved we should reactivate the step (erasing the continue-on-error directive)
This will avoid this errors:

Code formating
The code has to be well formatted following the formatting rules. For example in Python the code has to follow the PEP8.
Before create your PR, please make sure your code is formatted running:
$ >> black --exclude=venv,__pycache__ ./

@XaviTorello you can close https://github.com/GeeksCAT/anem-per-feina/issues/82 here by changing origin/master to master in the github action for mypy

Thanks @XaviTorello !


Is index.redb deterministic?

There has been a lot of discussion around sharing index.redb files. I think the official position of the project is "not your node, not your jpegs", and we want to encourage everyone to run ord with their own bitcoind, and verify everything themselves. However, there are many different security models that are possible with ord, and using a prebuilt index.redb file, might be an acceptable trade off.
We can make the trade offs even better if multiple independent developers can build index.redb files that have the same hash and sign them, which increases the difficulty of including malicious content (for example, something that exploited a bug in redb) to introduce a security vulnerability, or which tricks users about which inscriptions are where.
The first step in answering the question: Is index.redb deterministic? That is to say, when two people build an index.redb up to the same block height, with the same version of ord, is the index.redb file that is produced byte-for-byte compatible? If the answer is yes, we can start looking at how multiple devs can sign an index.redb. If the answer is no, we can try to look into why that is, and either modify our own use of redb, or write issues and PRs for redb that make it deterministic. I've been discussing this idea with @cberner on and off, and he has all the context on redb internals that would make this easy.

@veryordinally this might be interesting to you! A more safe first use-case could be building index.redb files for benchmarking, which would make that much easier.

There are no sources of randomness in redb. However, there are quite a few constraints if you want an exact byte for byte copy of the db (and I've never tested byte-for-byte compatibility :grinning: ). You would need to make every API call into redb, including open()/create() in exactly the same order. So in particular, building up to height M, stopping, then continuing to height N. Is not the same as building up to height N

Nice. Okay, so we would need to stop at a height, probably every 5000, because that's already the checkpoint number. I don't think there are any other sources of nondeterminism. If anyone wants to sync with --height-limit 100 and report the hash here, that would be awesome.

Running on master at f599b82:
ord --data-dir . --height-limit 10000 index
shasum -a 256 index.redb
8ace9fa062b267e6ee593c62a09c9eb6e18e9f7d05e0423182485849955ca8b2  index.redb

Unfortunately the hash changes if I delete index.redb and run again

I think you'll need to replace the HashMap caches in ord with a data structure with deterministic order, like a BtreeMap. For example, this looks like it will cause non deterministic behavior: https://github.com/casey/ord/blob/f599b82a93e90faa8937cc1ea7008261d457df50/src/index/updater.rs#L497


Replace tsyringe dependency injection

We're currently using tsyringe to provide all the endpoint methods in the client. I think it's time to remove it.
Is your feature request related to a problem? Please describe.
Bug
Describe the solution you'd like.
tsyringe is useful for dependency injection when dependencies are added to a "main" object. In our case, the client is merely a collector of various endpoint methods. In our case, tsyringe's abilities fall short. This bug emphasises the difference.
Removing tsyringe will reduce complexity, duplication and reduce the SDK's size. Especially after resolving https://github.com/spree/spree-storefront-api-v2-js-sdk/issues/85 with https://github.com/spree/spree-storefront-api-v2-js-sdk/pull/86. Instead of using tsyringe, let's instantiate endpoint classes directly and however we want.
Describe alternatives you've considered.

Reconfigure tsyringe to inject host value to all classes inheriting from Http. Using @inject on subclasses of Http worked but didn't work when trying to inject into Http. It seems classes don't inherit the runtime @inject decorator.
Have endpoint classes pull the host themselves. Couldn't figure out how to easily have them do that without using globals or some other indirect solution.

Will be released in 3.x.x - https://github.com/spree/spree-storefront-api-v2-js-sdk/pull/96.


Removing flask debugging

Changes
Flask was set to debug mode as default.
How to test

Check out this branch
Run the application and check everything works

Who can test
Anyone apart from @warrenbailey

LGTM 👍


Enable overrides in Land node

@liuly12 OK I've finished the example for how to do this in the WTW/WWTW node see: #70
We'll use this issue to discuss any additions required for the Land node override behaviour, which will of course be the most complex. Feel free to make a branch and PR when you're ready. Though possibly we will want to have multiple branches for the different surfaces - I'm easy.

@barneydobson Do you think self.data_input_dict should be revised as a property as well?

Good question, I was thinking about that in the context of Catchment. It's quite a specific format to allow someone to overwrite to?

In addition, if we do, it should probably run some checking to make sure the appropriate keys are included

I'd say don't put code for data_input_dict overrides for Land. It can be tackled in its own issue at a later date if necessary

Fixed with commit 31808b887b05b3bbe33398fe8c51b3414503d663

Add overrides to residence_times with commit 31808b8

@liuly12 think this should be on its own branch + PR

Following the discussion in https://github.com/ImperialCollegeLondon/wsi/pull/70/commits/82c74551df7b3b7241a151ab1afbda1694f3dd54#r1509275455. Still not very clear what would be the difference between using property and not using property. It seems to me that most overrides can be achieved without using property.
```
def apply_overrides(self, overrides = Dict[str, Any]):
self.surface = overrides.pop("surface",
self.surface)
self.area = overrides.pop("area",
self.area)
self.depth = overrides.pop("depth",
self.depth)
self.capacity = self.area * self.depth

So in this example, I agree that surface, area and depth should be set as above, but if capacity is a property:
@property
def capacity(self):
  return self.area * self.depth

Then when you try and set capacity it will throw an error. If we want a user to only ever define area and depth and never capacity - this is a handy way to force that and ensure that the value of capacity is always up to date. We will probably have to add something in __init__ that pings a deprecation message saying that capacity is now unused and that it is derived from depth and area to maintain compatibility with all the various config files we have.
In this sense - the rule is that we reserve @property for when the attribute is directly derived from other attributes of the class. Or (as with say percent_solids in WWTW) when that attribute requires us to recalculate something else (i.e., in this example process_parameters["volume"]["constant"])

I try to add the code below in the Surface
@property
def capacity(self):
  return self.area * self.depth

and then I get an error when initialising all the superclasses of Surface, using ImperviousSurface as an example:
    surface = ImperviousSurface(
  File "c:\users\leyan\documents\github\wsi\wsimod\nodes\land.py", line 599, in __init__
    super().__init__(depth=pore_depth, **kwargs)
  File "c:\users\leyan\documents\github\wsi\wsimod\nodes\land.py", line 366, in __init__
    super().__init__(capacity=capacity,
  File "c:\users\leyan\documents\github\wsi\wsimod\nodes\nodes.py", line 1136, in __init__
    Tank.__init__(self, **kwargs)
  File "c:\users\leyan\documents\github\wsi\wsimod\nodes\nodes.py", line 770, in __init__
    self.capacity = capacity
AttributeError: can't set attribute 'capacity'

I guess this error basically says that I'm making capacity as a property of Surface so that it cannot be initialised from its subclass `Tank' based on the current code?

OK it is clearly more complicated than expected - in which case just leaving it as is is probably more sensible and using your first solution:
self.capacity = self.area * self.depth


Address Sonar hint about regular expression optimization

https://sonarcloud.io/organizations/ksch-workflows/rules?open=java%3AS6397&rule_key=java%3AS6397


Explore cases where strings are returned and displayed to the user that may not pass through the localization engine

What needs to be done?
In a more complex application such as Backstage there's likely some cases where the UI initiates some action and receives a status or error message from the backend and displays this to the user.  When the app is fully localized it would be expected that these strings would also use the same language.

Does the localization support upstream also cater for strings that are returned from strictly backend code?  If so how is this handled?
If not, is there a plan to do this, either upstream, or are there other ways of tackling this?

part of #502

For this one I don't think it's required to translate every error message/string but if there's common errors that may be show to the user that come up regularly we should see what portion of the text may need translating.  For example if an error stack trace is passed along as a result from a request I wouldn't expect that to be translated, but if it's more like a  string containing the reason message for a HTTP 403/unauthorized response that makes it all the way to the UI, I would expect that should be translated.

This issue is a little nebulous as written, my intent is to just check if there's widespread cases where backend strings are used in error messages and if so should we consider running these through the translation engine.

Actually realized I put this as out of scope in the epic writeup and I yet to see widespread cases of this, so I think we can save this for when we decide to localize the backend and define it better for that effort.


See impact to economic benefit to the province

As an executive
I want to see the estimated capital investment in the province
so that I can make informed decisions
Acceptance Criteria
Scenario 1:
GIVEN ... (you or your condition)
WHEN ... (what you do)
THEN ... (what you see)
Scenario 2:
xxxxxx
Validation Rules?
(If yes, list here)
Screenshot
(Link above will always be the most up-to-date version.)

chart by team


Configuration should allow pyenv version to be hidden.

I regularly use both pyenv and virtualenv, but would prefer to not show the active pyenv version when a virtualenv is not active.

For some reason, my bullet-train prompt worked like that in my previous work computer. I have just installed bullet-train on a new one and it shows me 🐍system when virtualenv is not active. How can we disable this?

a solution is to manually open the theme file and edit it. Just comment this 2 lines
# elif which pyenv &> /dev/null; then
   # prompt_segment $BULLETTRAIN_VIRTUALENV_BG $BULLETTRAIN_VIRTUALENV_FG $BULLETTRAIN_VIRTUALENV_PREFIX" $(pyenv version | sed -e 's/ (set.*$//' | tr '\n' ' ' | sed 's/.$//')"


Decorate responses created by withRedirect helper

Currently the withRedirect helper returns the response implementation object instead of a newly decorated response.

Coverage increased (+0.003%) to 99.273% when pulling 9b3ceacdc13dfc779a5b38f2ce21810464fb55e3 on trvrnrth:decorate-with-redirect into 9ada50ce02593a498df2fdf246d63c361eeb0846 on slimphp:master.

I haven't added any tests to check that responses generated by all helpers are in fact decorated but I guess this might be something you want to consider too.

Looks good to me

Looks good to me, this PR should be merged before #78 though or just closed since #78 is being merged.


Opening multitree RaxML bootstrap file results in NAN tree with labels

Can you send me the tree file?  You can attach it here if you want.

Should be fixed now (1aca07e62d4ea66403f40e9721955639e78b085c).


Upgrade cluster version is not supported

I have issues
I'm submitting a...

[x] bug report
[ ] feature request
[ ] support request - read the FAQ first!
[ ] kudos, thank you, warm fuzzy

What is the current behavior?
I have provisioned an EKS cluster with Kubernetes version 1.18 successfully then I tried to update the Kubernetes version to 1.19 but when I am trying to run
terraform plan
I got the following error: Error: Your query returned no results. Please change your search criteria and try again.
If this is a bug, how to reproduce? Please include a code sample if relevant.
What's the expected behavior?
It should update the cluster kubernetes version same as updating it from the GUI
Are you able to fix this problem and submit a PR? Link here if you have already.
Environment details

Affected module version: v14.0.0
OS: Linux, Mac
Terraform version: v0.14.3

Any other relevant info

same with
Terraform v0.14.6

provider registry.terraform.io/hashicorp/aws v3.27.0
provider registry.terraform.io/hashicorp/helm v1.3.2
provider registry.terraform.io/hashicorp/kubernetes v2.0.2
provider registry.terraform.io/hashicorp/local v2.0.0
provider registry.terraform.io/hashicorp/null v3.0.0
provider registry.terraform.io/hashicorp/random v3.0.1
provider registry.terraform.io/hashicorp/template v2.2.0

This is related to #1141. As proposed by @dbirks, a workaround would be to set
worker_ami_name_filter_windows = "Windows_Server-2019-English-Core-EKS_Optimized-1.18-*"
even if you don't have any windows servers, at least until AWS releases the windows version to 1.19.
It worked fine for me. Thanks to @dbirks.

@italolelis work aour don't works for me. I'm using node_groups

I just opened https://github.com/terraform-aws-modules/terraform-aws-eks/pull/1371 to address this. Will you please review it and test it ?


Make Decimal formatting non-allocating

TryFormat(decimal ....) is implemented using CLR's ToString, which means it's allocating.
At some point we should implement an non-allocating version.

@AtsushiKan @shiftylogic

The plan for https://github.com/dotnet/corefx/issues/22403 is to add a non-allocating TryFormat methods.  That can be used instead of ToString.

@stephentoub, we need to support UTF8 buffers.

we need to support UTF8 buffers.

Sure, I'm not sure how my comment goes against that.
The PR added both UTF16 and UTF8 formatting.  For UTF16, the TryFormat method could be used to go directly into the caller-supplied output buffer; non-allocating.  For UTF8, the TryFormat method could be used to go into a stack-allocated Span<char>, since you know the longest a decimal can possibly be and it fits easily on the stack, and then convert that to the caller-supplied byte buffer; again, non-allocating (and no more conversion than is currently done in your PR).
This gets you to the goal of being non-allocating, which is what this issue is about, hence my comment.  If your goal is to format directly in UTF8 and skip UTF16, then yes, you'd need a custom implementation, but I have to imagine that if getting a UTF8 TryFormat for decimal to be non-allocating/non-extra-converting is lower priority (as mentioned in the PR), it'd be even lower priority to get one that's non-allocating but still does the conversion. :)

@stephentoub, we also want to support existing versions of .net.

we also want to support existing versions of .net.

In which case we can multi-target and the build against .NET Core can use the newer APIs.

In which case we can multi-target/cross-compile and the build against .NET Core can use the newer APIs.

What would it buy us? It would just make the packages more complicated (i.e. cross-compiled)

What would it buy us?

The same thing it generally buys us: not having to re-implement the same functionality, or even if we choose to implement that functionality, getting the benefit of a non-allocating version until we do.  I'm not sure why this is controversial: this whole thread started because you said it's allocating due to using ToString, and all I suggested was being able to use TryFormat instead once it's there.

I'm not sure why this is controversial: this whole thread started because you said it's allocating due to using ToString, and all I suggested was being able to use TryFormat instead once it's there.

It's controversial because: a) today the package is not-cross compiled, i.e. simple. b) decimal is really here for completeness, i.e. I kind of don't care too much that it's allocating. Note that the rest of the parser are non-allocating already.

today the package is not-cross compile

Right, this is the case today but it is unlikely to be the case once it moves to corefx. All similar packages got cross-compiled very quickly as soon as they are moved to corefx. This one is not going to be exception - folks will want to make it run better on .NET Core using constructs that are available in .NET Core only.
Also, I do not think we will want to have the optimized parsing routines duplicated in .NET Core. We will want to use the same optimized core of parsing, formatting and encoding routines for both existing classic methods and these new methods. It means that this code is very likely going to live in .CoreLib on .NET Core once we are done.

I think this is done.


implement urlquery

Implement urlquery

Codecov Report

Merging #45 (77ab105) into main (b6e7c50) will increase coverage by 1.27%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main      #45      +/-   ##
==========================================
+ Coverage   44.68%   45.95%   +1.27%
==========================================
  Files          32       32
  Lines        1551     1558       +7
==========================================
+ Hits          693      716      +23
+ Misses        858      842      -16

Impacted Files
Coverage Δ

src/builtins/impls/mod.rs
0.00% <ø> (ø)

src/builtins/mod.rs
94.04% <ø> (ø)

src/builtins/impls/urlquery.rs
100.00% <100.00%> (+100.00%)
:arrow_up:

tests/smoke_test.rs
100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@sandhose any option that you can review this PR? I did all your changes.

@kaplanelad Very sorry about the delay. I finally manage to find some time to work on the crate recently, which I wasn't able before.
I went ahead and rebased your branch and did the following changes:

define a OneOrMany<T> type, which can hold either a T or a Vec<T>. This avoids having to default to an empty string when the object has something else than a string.
uses BTreeMap instead of HashMap, because they are sorted by default, so it keeps the output stable in decode_object and avoids having to sort the input in encode_object.


⚠ Bofrost.at has degraded performance

In 25f1aaf, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 2171 ms

Resolved: Bofrost.at performance has improved in dea6513 after 304 days, 10 hours, 0 minutes.


Index page url for app

Let's assume we use nginx...
Whats the best url? e.g.
kiteyttaja.qatsi.info
qatsi.info/kiteyttaja
Url should be chosen beforehand, so we might have some fiddling time if the url is sent along with the game packages at next week or so.

I think this is good: kiteyttaja.qatsi.info

Alright yeah, I am unsure do people trust "subdomain naming style", but let's assume that it's okay. I guess I should put a more relevant placeholder site for qatsi.info as well, if users check it?

kiteyttaja.qatsi.info it is.


Doc builds on master are broken

It looks like something is going wrong with generating the PDF version

Caused by https://github.com/spatialaudio/nbsphinx/issues/584


Allow the use of STS generated credentials

We make use of linked accounts and pretty strictly disallow the creation of "local" IAM users, so we really need to make use of the sts assume-role capability which results in:
SecretAccessKey,
SessionToken,
AccessKeyId
Doesn't look like I can currently make use of this approach since src/bosh_aws_cpi/lib/cloud/aws/cloud.rb does a Aws::Credentials.new(access_key, secret_access_key) call. The library should support using the session token as well, per:
http://docs.aws.amazon.com/sdkforruby/api/Aws/Credentials.html
Not sure what the operational implications are though, so maybe not the best approach.

@jrrickard would iam instance profiles help you? http://bosh.io/docs/aws-iam-instance-profiles.html

Greetings!
I am trying to set up a bosh director in AWS and I am getting the following error
was not able to validate the provided access credentials.
It seems that we have enabled the MFA in AWS, which blocks many calls unless is MFA authenticated.  I had to do something like that in order to make terraform work
export AWS_MFA_Device=XXX
function aws-mfa() {
  unset AWS_SECRET_ACCESS_KEY; unset AWS_ACCESS_KEY_ID
  ret=$(aws sts get-session-token --serial-number $AWS_MFA_Device --token-code $1)
  export AWS_ACCESS_KEY_ID=$(echo $ret| jq -r '.Credentials.AccessKeyId')
  export AWS_SESSION_TOKEN=$(echo $ret| jq -r '.Credentials.SessionToken')
  export AWS_SECRET_ACCESS_KEY=$(echo $ret| jq -r '.Credentials.SecretAccessKey')
}

Given that the cpi does not accept the session token, i think it is the same issue?
What do you think? Any plan to schedule it?
Thanks

@karampok The maximum time for a session token is 36 hours. You would have to update the token code every 36 hours in order for the aws client to retrieve a new valid token.
Is that a desirable use case for you?

Hi, for my development use case this is the only option. I have only access to STS generated credentials.  Will something stop working if the aws client has expired credentials?

I think this is a quite important and necessary change.
Using STS temporary credentials is a recommended practice, and usually enforced everywhere. It is used not only for security, but also to implement delegation using roles and permissions.
The instances profiles is one way of getting temporary credentials from one machine.
But there are multiple cases that  you have to use environment variables. For instance, when using locally the CPI for when deploying a new environment using bosh create-env: The user must would assume role using MFA, and then deploy the application.
In some places they even block using the metadata endpoint to get instance profile credentials, as it might be a security thread when running containerised environments (e.g. running an automated deployment on concourse).
Adding support for this should be a small and risk-less change, it would allow the CPI fully support AWS auth mechanism and unblock users to use.

Regarding the implementation, I would say this should be implemented as follows:

add the optional param aws_session_token, to be used together with access_key_id ,  secret_access_key when calling http://docs.aws.amazon.com/sdkforruby/api/Aws/Credentials.html#initialize-instance_method in https://github.com/cloudfoundry-incubator/bosh-aws-cpi-release/blob/703149c7f83bac29427995299d8c19a0d9c04557/src/bosh_aws_cpi/lib/cloud/aws/config.rb#L32. If  session_token is not provided, we pass nil.

Extend the logic in https://github.com/cloudfoundry-incubator/bosh-aws-cpi-release/blob/703149c7f83bac29427995299d8c19a0d9c04557/src/bosh_aws_cpi/lib/cloud/aws/config.rb#L34 so that it would check if the environment variables are set. If they are, it would use them, otherwise it would try with instance profile.

I am not sure how the official ruby library deals with the credential in environment variables, but it theory it does: http://docs.aws.amazon.com/sdkforruby/api/. Maybe we should have a look.
This approach would cover all the use cases:

The option 1, would allow pass the credentials and variables during interpolation: bosh create-env -v session_token=${AWS_SESSION_TOKEN} -v access_key_id=${AWS_ACCESS_KEY_ID}  and -v  secret_access_key=${AWS_SECRET_KEY}. This is great to use in automated scripts with bosh-lite

The option 2 would allow define environment variables in actual deployments or other uses of the CPI.

available in v68.


Allow create object with non-default constructor in SystemPool

Problem
What is being fixed?  Examples:

Currently, there are some limitations to put an Object into the SystemPool, the Object need to have a default constructor defined, also a separate init function to initialize the Object. This limits the usage of SystemPool to only those classes which meets those pre-conditions.

We need to Allow create object with non-default constructor in SystemPool,  so  a typical usage will look like:

     *           class Foo: public Object {
     *              Foo(int a, int b, int c)
     *           ...}
     *           ....
     *           Foo foo;
     *           foo.TryCreate(a, b, c);

Change overview
Allow create object with non-default constructor in SystemPool
Testing
How was this tested? (at least one bullet point required)

covered by unit tests added

I have tried to dynamically implementing the pool by switching the real object and a placeholder object (contains a mRefCount). It requires that the object fulfills:

is_trivially_constructible: this is required because we want to skip to constructor of T to preserve value of mRefCount
is_trivially_destructible: this is required because we want to skip to destructor, as the mRefCount already went to 0, so we are not allowed to touch the object, otherwise it will cause a memory racing. (Yes, the racing problem already exists in the code)
is_standard_layout and offsetof(Object, mRefCount) == offsetof(T, mRefCount): this is required such that the memory location of mRefCount is same for T and the placeholder object.

It is nearly impossible to meet the requirement, so I suggest using a bitmap instead.

It looks all solutions we explored here are either involuted or hacky.
Take a step back to look at this requirement, triggering the object construction outside of its constructor is not a standard way to initialize an object. Doing some investigations on-line, the recommended practice to add a object to a static pool are:

Construct the object first and move the object to the static pool, or
Allocate the object from the static pool and call init function of object to initialize the object

So it ends up we just need to make sure all objects created from the system pool have a default constructor and init function to initialize itself.

How does this PR relate to #9590 and #9748?

I have had a chat with Yufeng.
The main concern of this PR is that it is to hacky. Yufeng has the concern that using an extra bitmap along with the pool is not memory efficient. First I want to analyze the  memory usage of #9590

How does this PR relate to #9590 and #9748?

I have had a chat with Yufeng.
The main concern of this PR is that it is to hacky. Yufeng has the concern that using an extra bitmap along with the pool is not memory efficient. First I want to analyze the memory usage of #9748

The overhead is 1 bit per allocated object.  That seems like a very modest cost to have correct and safe code.  Unless I'm misunderstanding, it seems like #9590 and #9748 are the way to go and we should get through those reviews, merge those and close this (#9505).

How does this PR relate to #9590 and #9748?

I have had a chat with Yufeng.
The main concern of this PR is that it is to hacky. Yufeng has the concern that using an extra bitmap along with the pool is not memory efficient. First I want to analyze the memory usage of #9748

The overhead is 1 bit per allocated object. That seems like a very modest cost to have correct and safe code. Unless I'm misunderstanding, it seems like #9590 and #9748 are the way to go and we should get through those reviews, merge those and close this (#9505).

Agree with this path.


[hotfix] fix missing jar streampark-flink-shims_flink.

[hotfix] fix missing jar streampark-flink-shims_flink.

assembly excute twice, I will carry on to fix it.

Before:

After:


Fix phpunit deprecation

Fix phpunit deprecations
root@4a1e0c585181:/var/www/html# ./vendor/bin/phpunit
PHPUnit 9.5.20 #StandWithUkraine

.........W..W..................................................  63 / 169 ( 37%)
............................................................... 126 / 169 ( 74%)
...........................................                     169 / 169 (100%)

Time: 00:15.272, Memory: 40.50 MB

There were 2 warnings:

1) App\Tests\Api\Status\GitHubStatusApiTest::testSetIssueStatusRemovesExcessStatuses
The at() matcher has been deprecated. It will be removed in PHPUnit 10. Please refactor your test to not rely on the order in which methods are invoked.

2) App\Tests\Api\Status\GitHubStatusApiTest::testSetIssueStatusRemovesUnconfirmedWhenBugIsReviewed
The at() matcher has been deprecated. It will be removed in PHPUnit 10. Please refactor your test to not rely on the order in which methods are invoked.

Thanks Antoine!


Optimize OP2 rewrites if both arguments are NUM

If P1 of an OP2 node is a NUM value, go ahead and do the op without creating an OP1 reduction.
This PR is for documentation purposes. This optimization has little effect on performance regression tests. On the radix sort example, it does eliminate a significant number of OP rewrites, but the overall execution time is not improved much at all.

I just did a test on radix_sort_lam and this patch did have a measurable improvement.

On second thought, go ahead and re-implement it and add the counter update. I'll start on the native calls PR


Drop Python 2 support

This builds on #24 to drop support for Python 2, when we're ready for that.

I pushed fixup commits to drop support for 3.3 as well, since I use enums in #21 and they were only added in 3.4. (And I'm kind of hesitant about supporting versions that aren't tested by the continuous integration script anyway) I figured we might as well do that at the same time as dropping Python 2 support.
Happy to abandon those commits and keep support for 3.3 if anyone wants to argue for it, though.

@coordt you mind giving a final sanity check?


Meshes with dynamic lighting lose ambient lighting on edges of LightmapGI bounds

Tested versions

Reproducible in v4.2.1.stable.official [b09f793f5], v4.1.3.stable.official [f06b6836a], and v4.0.4.stable.official [fc0b241c9]

System information
Godot v4.2.1.stable - Windows 10.0.22621 - Vulkan (Forward+) - dedicated AMD Radeon RX 6700 XT (Advanced Micro Devices, Inc.; 27.20.21034.37) - AMD Ryzen 7 5700X 8-Core Processor (16 Threads)
Issue description
When a dynamically lit mesh moves into the bounds of a LightmapGI, it completely lose ambient lighting (turns black) around the edge and returns to normal after completely going into or out of the edges.
https://github.com/godotengine/godot/assets/25323231/6874f7a4-87e7-4968-ad1c-caa8037d39a7
Steps to reproduce

Set up a scene with lightmap GI
Create a mesh with dynamic GI
Move it in and out of the LightmapGI's bounds as indicated by the generated lightmap probes

Minimal reproduction project (MRP)
Lightmap.zip

My guess is that the time-based interpolation starts with purely black samples, so it gradually fades from black when a dynamic object enters the lightmap's bounds. It should probably initialize itself with the first sample found.
In the meantime, you can add LightmapProbe nodes in each corner of the level so that dynamic object lighting extends beyond the static meshes. Automatic probe generation will adapt itself to add probes in these areas if they're large enough.

You can confirm this theory by increasing the Rendering > Lightmapping > Probe Capture > Update Speed setting to a higher value.

The "black area" is actually very defined, and moving the object in that area does not fade it to lighter ambient colors.

Hi, I have created a small project containing several test scenes to verify the correct generation and display of LightmapGI and LightmapProbes.
Godot LightmapGI Tests
The scene that tests dynamic objects is located in Test 02 scene.

In addition to the issue of objects going outside the Probes area, there is another one related to the fact that dynamic objects are illuminated entirely with only one Probe, even if their size causes their parts to be in differently illuminated locations.

cts are illuminated entirely with only one Probe, even if their size causes their parts to be in differently illuminated locations.

How do other engines behave in similar conditions? I think sampling from multiple probes could impact performance considerably. We might need to also consider probe density if we want to have that as an option in the engine.

OK, so to test this I installed Unity (v2018 because the later ones refuse to log into the license server for some reason) and I have to admit that I am now very disappointed that lighting dynamic objects works in a similar way. Regardless of the size, the moment the center of the object comes within range of the Lightprobes, the whole object starts receiving additional lighting from them.
In the example below, all objects except the cylinder are marked as static. The light "sun" and blue light in the box also only affect the lightmaps. One lightprobe placed in the box receives blue light directly.

When the cylinder is lowered so that it comes "within range" of the blue lightprobe, it turns blue.

On the plus side for Unity, it should be noted that the change of light when entering the range of lightprobes is smooth and there are no sudden changes as in Godot currently.
Unity also tried to fix the problem by introducing Light Probe Proxy Volume, where a selected dynamic object with such a component added has the ability to interpolate lighting between lightprobes.
https://docs.unity3d.com/Manual/class-LightProbeProxyVolume.html

If you want a dynamic object to receive lighting from multiple lightmap probes, you have to split it into several smaller meshes.

Dynamic objects losing ambient light at the edge is caused by the fact that the object is assigned to the lightmap the moment its AABB intersects with the AABB of the lightmap, but then when the renderer goes to find the appropriate tetrahedron of probes to use, it bails out early if the center of the object is not inside the lightmap.
The area between where the edge and the center of the object intersects with the lightmap is a no man's land.
https://github.com/godotengine/godot/blob/aef11a14274f6f9e74ad91ead1d7c07ea1dd7f5f/servers/rendering/renderer_scene_cull.cpp#L2029-L2036

@permelin given your experience with lightprobes (https://github.com/godotengine/godot/pull/90701 https://github.com/godotengine/godot/pull/90702), will you also try to address this problem?

It seems the fix would be fairly simple: instead of checking just one point in the bounds, check for all points from the AABB and only continue; if none of them is within the bounds.

@permelin given your experience with lightprobes (https://github.com/godotengine/godot/pull/90701 https://github.com/godotengine/godot/pull/90702), will you also try to address this problem?

I hadn't planned to do so.

It seems the fix would be fairly simple: instead of checking just one point in the bounds, check for all points from the AABB and only continue; if none of them is within the bounds.

I think you could just remove this check. IIRC, this code path will not be hit unless the lightmap has been assigned to the object, and that will only happen if the lightmap and object already intersect. Quitting early is redundant.
The real problem is the blend factor a bit further down. If the center of the instance AABB is not inside the lightmap, the blend will be zero, which means no light anyway. So that algorithm must be changed.
Again, if I remember correctly, the blend is actually there to handle overlapping lightmaps. It's possible that a solution would simply be to check if there is only a single lightmap assigned to the object and if so, set the blend to 1. (Though a perfect solution would blend at the bounds with whatever ambient/indirect light is outside the lightmap for a seamless transition.)


Add links between classic and new UIs

The links preserve the graph settings between the two UIs.
Fixes https://github.com/prometheus/prometheus/issues/6162
Signed-off-by: Julius Volz<EMAIL_ADDRESS>

Screenshots:
Classic UI:

New UI:


Fix reading tag with newline

Works for the following xml:
<?xml version="1.0"?>
<note xmlns="http://www.w3schools.com"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.w3schools.com note.xsd">
  <to>Tove</to>
  <from>Jani</from>
  <heading>Reminder</heading>
  Don't forget me this weekend!
</note>

but doesn't work for the following xml (newline right after tag)
<?xml version="1.0"?>
<note
xmlns="http://www.w3schools.com"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.w3schools.com note.xsd">
  <to>Tove</to>
  <from>Jani</from>
  <heading>Reminder</heading>
  Don't forget me this weekend!
</note>

This PR fix this situation
fix #120

@HyukjinKwon merge it plz

I find you as people in databricks
@andrewmchen
@falaki
@helfer
@jcdavis
@jkbradley
@sadikovi
@WeichenXu123
@yogeshg
I need review of 2 people and after merge it

Hi @BioQwer none of those people work on this project, and I wouldn't tell people to merge it. I may have some time to look later.

Hi @BioQwer none of those people work on this project, and I wouldn't tell people to merge it. I may have some time to look later.

@srowen thanks, wait for review.


Datatables is removing classes from hidden rows and cells.

We have classes assigned to certain td's so that we can manipulate them as the user interacts with our application. Unfortunately it looks like Datatables is stripping out the classes from the rows and cells that are being hidden (for responsive mode).
I tried to use a Datatable Render from here:
https://datatables.net/reference/option/responsive.details.renderer
But that doesn't seem to allow me to maintain the td classes, only replace them with "default" classes.
This seems like a really bad way to handle rendering hidden rows. I guess I don't understand why you wouldn't just keep the existing classes and add the childelement class.
Am I just missing something? Is there a way to keep the classes?

Apologies if the "New issue" template text isn't clear. Could you post question in the DataTables forums please, with a link to a test case showing the issue.
From what I understand here, you have a cell with class names applied to it, but are wondering why that class name is not used in the responsive view. Is that correct? If so, that is because the responsive view doesn't use the td cell from the DataTable - by default it uses a ul/li list and a copy of the data. There is currently no option to have Responsive use the original cell from the table (I did try that before, but found it introduced a lot of complexity and resolved to go back to it when I had time to do so).

You are correct about the situation.
Instead of using the original cell would it be possible to just capture the current classes, id, and data-* and pass them into the newly rendered ul/li?
I did find a minor workaround in that I can use the details.renderer to push the title of the column into the class of the cell. Not ideal but it's a work around.
I think this feature would make a lot of sense.

Copying classes sounds like a good idea to allow styling to be maintained - thanks for the suggestion. I've opened an issue for this in the Responsive repo.
I'm less keen on copying the ID - the way Responsive works is to hide the cells with display:none, so the cells are still present in the DOM, with their id. Copying the id to another element would mean that there were two elements with the same id, which isn't valid.

Can I also suggest that you allow the classes to be pushed into the details.renderer so that not only can you still capture the original class but you could also custom render the hidden areas and put back that class. So in our situation we found that we could better control the alignment and layout by using the renderer to render the spans as td's. In our situation I'd love to be able to insert a col.classes into the td element.
I think this minor change could really help open up more interactions inside of the datatables hidden rows.


Clipping and blending: blend modes and pen support

This PR:

Deprecates and hides additive blending blocks
Adds blocks for working with blend modes (https://github.com/TurboWarp/extensions/pull/263)

Makes this extension affect pen (https://github.com/TurboWarp/extensions/issues/302)

This looks incredible!!!

how do i load this onto the editor? i know that the raw file url shouldn't work, but the url for the main file doesn't work either so which link do i paste into the custom extensions box?

how do i load this onto the editor? i know that the raw file url shouldn't work, but the url for the main file doesn't work either so which link do i paste into the custom extensions box?

Neither, as only files on extensions.turbowarp.org and localhost:8000 are loaded unsandboxed. You'll have to wait for this PR to be merged. (However, on the packager, all extensions are loaded unsandboxed.)


Shelf header now wraps to a second line.

[x ] I have read the FAQ.

Describe the bug
Shelf header now wraps to a second line.
Versions:

Salt Rim: v3.6.0
Bar Assistant: v4.2.4

Would like this to resize down to a single row of stats:

Hello, not a bug, it was made with intent to wrap to second line if needed.

OK, not a bug, design issue maybe? There isn't a way to make it one line.  Making the window bigger doesn't help, reducing the font size doesn't help.

I get what you mean but not an issue. There will be possibly more stats in the future, and the line break heavily depends on selected language and available numbers


ModuleNotFoundError: No module named 'exllamav2'

The test environment: the server OS is Ubuntu 22.04 and there is a T4 GPU available.
(1)did below install and infer steps according to https://huggingface.co/TigerResearch/tigerbot-13b-chat-4bit-exl2:
conda create --name tigerbot python=3.8
conda activate tigerbot
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia
git clone https://github.com/TigerResearch/TigerBot
cd TigerBot
pip install -r requirements.txt
git clone https://github.com/turboderp/exllamav2
cd exllamav2
pip install -r requirements.txt
cd ..
CUDA_VISIBLE_DEVICES=0 python other_infer/exllamav2_hf_infer.py --model_path TigerResearch/tigerbot-13b-chat-4bit-exl2
(2)Got the error:
Traceback (most recent call last):
File "other_infer/exllamav2_hf_infer.py", line 9, in
from exllamav2 import ExLlamaV2, ExLlamaV2Cache, ExLlamaV2Config
ModuleNotFoundError: No module named 'exllamav2'
(3)Please advise how to fix this issue.

try "pip install exllamav2"? and "pip show exllamav2" to check if installed successfully

(1)Did:
pip install exllamav2
CUDA_VISIBLE_DEVICES=0 python other_infer/exllamav2_hf_infer.py --model_path TigerResearch/tigerbot-13b-chat-4bit-exl2
(2)Got:
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda-12.1'
Traceback (most recent call last):
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/exllamav2/ext.py", line 14, in
import exllamav2_ext
ModuleNotFoundError: No module named 'exllamav2_ext'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "other_infer/exllamav2_hf_infer.py", line 9, in
from exllamav2 import ExLlamaV2, ExLlamaV2Cache, ExLlamaV2Config
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/exllamav2/init.py", line 3, in
from exllamav2.model import ExLlamaV2
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/exllamav2/model.py", line 12, in
from exllamav2.linear import ExLlamaV2Linear
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/exllamav2/linear.py", line 5, in
from exllamav2 import ext
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/exllamav2/ext.py", line 121, in
exllamav2_ext = load
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1308, in load
return _jit_compile(
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1710, in _jit_compile
_write_ninja_file_and_build_library(
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1810, in _write_ninja_file_and_build_library
_write_ninja_file_to_build_library(
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 2199, in _write_ninja_file_to_build_library
cuda_flags = common_cflags + COMMON_NVCC_FLAGS + _get_cuda_arch_flags()
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1980, in _get_cuda_arch_flags
arch_list[-1] += '+PTX'
IndexError: list index out of range

Hello, you can upgrade your Python version to 3.9 first and try again. We work normally with python=3.10 and CUDA version=11.6. The recommended Python version is >=3.9. We will also investigate the solution as soon as possible.

I think I found the reason. When switched to Python 3.8, I encountered the same problem as you. I realized that it might be due to a mismatch between the torch version and the CUDA version. Therefore, I reinstalled torch under Python 3.8 and ensured that it was consistent with the current CUDA version. The model loaded smoothly. Good Luck !

conda create --name tigerbot python=3.8
conda activate tigerbot
#pytorch official installation
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia
git clone https://github.com/TigerResearch/TigerBot
cd TigerBot
pip install -r requirements.txt
git clone https://github.com/turboderp/exllamav2
cd exllamav2
pip install -r requirements.txt
pip install exllamav2
cd ..
CUDA_VISIBLE_DEVICES=0 python other_infer/exllamav2_hf_infer.py --model_path TigerResearch/tigerbot-13b-chat-4bit-exl2
loading model: TigerResearch/tigerbot-13b-chat-4bit-exl2...
Traceback (most recent call last):
File "other_infer/exllamav2_hf_infer.py", line 300, in
fire.Fire(main)
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/fire/core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "other_infer/exllamav2_hf_infer.py", line 190, in main
model = get_model(model_path)
File "other_infer/exllamav2_hf_infer.py", line 177, in get_model
model = Exllamav2HF.from_pretrained(model)
File "other_infer/exllamav2_hf_infer.py", line 161, in from_pretrained
config.prepare()
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/exllamav2/config.py", line 52, in prepare
assert os.path.exists(self.model_dir), "Can't find " + self.model_dir
AssertionError: Can't find TigerResearch/tigerbot-13b-chat-4bit-exl2

pip install torch==2.0.1+cu117 torchvision torchaudio --extra-index-url https://mirror.sjtu.edu.cn/pytorch-wheels/cu117

yes. installed torch==2.0.1+cu117.

yes, it's a bug. exllamav2 will first check local path. You can download model weights from HF manually. We will repair it soon.

Thank you! Another option is to do 4 bits quantization with AutoGPTQ only.


fix: Update _index.md links

- What I did
Fixed the button links
- How I did it
Typed the correct links
- How to verify it
You can clone my fork and then 'npm-run-start' on your local to check if the changes work
- Description for the changelog

closes #162

Hey @AyushK-101, thanks for the PR!
I noticed you added your "closes #xxx" tag inside the comment:
<!--
Please make sure you've read and understood our contributing guidelines in CONTRIBUTING.md

If this is a bug fix, make sure your description includes "fixes #xxxx", or
"closes #xxxx"
closes #162
Please provide the following information:
-->

I've gone ahead and moved it outside the comment for you, but just so you know for in the future:
<!-- #STUFF HERE# --> is an html comment tag

@AyushK-101
Hey! Thank you for your Hacktoberfest contribution. You could be eligible for a prize.
To be eligible, please send an email to<EMAIL_ADDRESS>with a screenshot of the PR approval email (or just forward the PR approval email to us). This is to prove that your email corresponds to the correct GitHub account. Thank you!
When it comes time to distribute the prizes, we will look at all the emails we've received from that email so don't forget to do this!


Create a new pull request by comparing

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

25 of 25 (100.0%)  changed or added relevant lines in 3 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.4%) to 95.927%

Totals

Change from base Build<PHONE_NUMBER>:
0.4%

Covered Lines:
2030

Relevant Lines:
2081

💛  - Coveralls


Add ReadinessEndpoints to Proxy API

Updates Proxy spec with readinessEndpoints, a list of endpoints used to verify proxy readiness before updating status fields.
Jira: NE-206
@derekwaynecarr @mrunalp @bparees PTAL

if we ignore the proxy.spec value because it failed validation (due to the health check or any other reason) how does the admin become aware that we rejected their proxy config, other than by looking at "status" and seeing that their desired spec values are not reflected there?
And how do they know why their spec was rejected?
Do we need some additional status fields for error reporting?  or is there some other mechanism by which we'll be exposing this information?

how does the admin become aware that we rejected their proxy config

@bparees how about adding status conditions?

@bparees how about adding status conditions?

I think it's a reasonable first step (we may want to do more to get those conditions in front of the admin when there are issues, but having them on the resource is a necessary/appropriate starting point imho)

@bparees I'm glad that you feel status conditions are a good initial step. I have updated the PR accordingly, PTAL.

@bparees I'm glad that you feel status conditions are a good initial step. I have updated the PR accordingly, PTAL.

sounds like things went a different direction on the slack thread (having the admission controller do all the checks up front and completely reject the spec update if it fails), so i guess we won't need status if that is the approach.  Sorry for leading you astray.

@bparees I removed status conditions.

/lgtm

/approve