Demo code hangs at write on ESP32

Hi,
I'm using a Lilygo Lora32 T3 1.6.1 board with an ESP32 chip. I'm not sure what I'm doing wrong, but the write operations block forever.
The command I used to start the demo: cargo +esp run --release --example demo --features esp32 --target xtensa-esp32-none-elf
I tried with new esp-hal versions, with a pull request to disable the other core, and I originally reproduced this while using embassy.
The output:
Chip type:         esp32 (revision v1.1)
Crystal frequency: 40MHz
Flash size:        4MB
Features:          WiFi, BT, Dual Core, 240MHz, Embedded Flash, Coding Scheme None
MAC address:       64:b7:08:ad:1d:54
App/part. size:    92,336/4,128,768 bytes, 2.24%
[00:00:01] [========================================]      17/17      0x1000
[00:00:00] [========================================]       1/1       0x8000
[00:00:02] [========================================]      27/27      0x10000
[2023-11-11T17:55:59Z INFO ] Flashing has completed!
Commands:
    CTRL+R    Reset chip
    CTRL+C    Exit

ets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 188777542, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:7104
0x3fff0030 - _stack_end_cpu0
    at ??:??
load:0x40078000,len:15576
0x40078000 - ets_delay_us
    at ??:??
load:0x40080400,len:4
0x40080400 - _init_end
    at ??:??
ho 8 tail 4 room 4
load:0x40080404,len:3876
0x40080404 - _init_end
    at ??:??
entry 0x4008064c
0x4008064c - __user_exception
    at ??:??
I (31) boot: ESP-IDF v5.1-beta1-378-gea5e0ff298-dirt 2nd stage bootloader
I (31) boot: compile time Jun  7 2023 07:48:23
I (34) boot: Multicore bootloader
I (38) boot: chip revision: v1.1
I (42) boot.esp32: SPI Speed      : 40MHz
I (46) boot.esp32: SPI Mode       : DIO
I (51) boot.esp32: SPI Flash Size : 4MB
I (55) boot: Enabling RNG early entropy source...
I (61) boot: Partition Table:
I (64) boot: ## Label            Usage          Type ST Offset   Length
I (72) boot:  0 nvs              WiFi data        01 02 00009000 00006000
I (79) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (86) boot:  2 factory          factory app      00 00 00010000 003f0000
I (94) boot: End of partition table
I (98) esp_image: segment 0: paddr=00010020 vaddr=3f400020 size=0292ch ( 10540) map
I (110) esp_image: segment 1: paddr=00012954 vaddr=40080000 size=022bch (  8892) load
I (119) esp_image: segment 2: paddr=00014c18 vaddr=00000000 size=0b400h ( 46080)
I (140) esp_image: segment 3: paddr=00020020 vaddr=400d0020 size=06864h ( 26724) map
I (151) boot: Loaded app from partition at offset 0x10000
I (151) boot: Disabling RNG early entropy source...
Flash size = 4194304
Read from 129000:  [70, a7, 20, 25, ec, d3, 16, 3a, 01, 42, 26, 00, 5a, 54, 72, 45, 18, c0, 20, 00, c0, 20, 00, 06, e9, ff, 00, 00, 00, 00, 17, 40]

The address it is reading from / writing to looks a bit weird in your logs: Read from 129000 - it should use the NVM partition (the demo hardcodes the address to 0x9000

I can confirm that the write method hangs, even when writing to my own partition. I guess the bytes.is_empty() condition never becomes true for some reason.

On what chip exactly? (including revision) For all chips except ESP32 we just call the ROM functions, On ESP32 unfortunately we have to do a bit more (but not that much more)

On what chip exactly? (including revision) For all chips except ESP32 we just call the ROM functions, On ESP32 unfortunately we have to do a bit more (but not that much more)

nik@frifot ~/S/s/hazelos (main)> cargo espflash board-info
[2023-11-15T10:43:03Z INFO ] Serial port: '/dev/ttyUSB0'
[2023-11-15T10:43:03Z INFO ] Connecting...
[2023-11-15T10:43:04Z INFO ] Using flash stub
Chip type:         esp32 (revision v1.0)
Crystal frequency: 40MHz
Flash size:        4MB
Features:          WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
MAC address:       c8:c9:a3:c9:a7:ac

Thanks - looks like the same I have here:
❯ cargo +esp run --release --example demo --features esp32 --target xtensa-esp32-none-elf
    Finished release [optimized] target(s) in 1.46s
     Running `espflash flash --monitor target\xtensa-esp32-none-elf\release\examples\demo`
[2023-11-15T10:50:57Z INFO ] Serial port: 'COM4'
[2023-11-15T10:50:57Z INFO ] Connecting...
[2023-11-15T10:50:57Z INFO ] Using flash stub
[2023-11-15T10:50:57Z WARN ] Setting baud rate higher than 115,200 can cause issues
Chip type:         esp32 (revision v1.0)
Crystal frequency: 40MHz
Flash size:        4MB
Features:          WiFi, BT, Dual Core, 240MHz, Coding Scheme None
MAC address:       cc:50:e3:99:49:e8
App/part. size:    92,400/4,128,768 bytes, 2.24%
[00:00:00] [========================================]      17/17      0x1000
[00:00:00] [========================================]       1/1       0x8000
[00:00:00] [========================================]      26/26      0x10000
[2023-11-15T10:50:59Z INFO ] Flashing has completed!
Commands:
    CTRL+R    Reset chip
    CTRL+C    Exit

��������������������Ҫ�j

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:7104
0x3fff0030 - _stack_end_cpu0
    at ??:??
load:0x40078000,len:15576
0x40078000 - ets_delay_us
    at ??:??
load:0x40080400,len:4
0x40080400 - _init_end
    at ??:??
ho 8 tail 4 room 4
load:0x40080404,len:3876
0x40080404 - _init_end
    at ??:??
entry 0x4008064c
0x4008064c - __user_exception
    at ??:??
I (30) boot: ESP-IDF v5.1-beta1-378-gea5e0ff298-dirt 2nd stage bootloader
I (31) boot: compile time Jun  7 2023 07:48:23
I (33) boot: Multicore bootloader
I (37) boot: chip revision: v1.0
I (41) boot.esp32: SPI Speed      : 40MHz
I (45) boot.esp32: SPI Mode       : DIO
I (50) boot.esp32: SPI Flash Size : 4MB
I (54) boot: Enabling RNG early entropy source...
I (60) boot: Partition Table:
I (63) boot: ## Label            Usage          Type ST Offset   Length
I (71) boot:  0 nvs              WiFi data        01 02 00009000 00006000
I (78) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (86) boot:  2 factory          factory app      00 00 00010000 003f0000
I (93) boot: End of partition table
I (97) esp_image: segment 0: paddr=00010020 vaddr=3f400020 size=0292ch ( 10540) map
I (110) esp_image: segment 1: paddr=00012954 vaddr=40080000 size=01fa8h (  8104) load
I (118) esp_image: segment 2: paddr=00014904 vaddr=00000000 size=0b714h ( 46868)
I (139) esp_image: segment 3: paddr=00020020 vaddr=400d0020 size=068a0h ( 26784) map
I (150) boot: Loaded app from partition at offset 0x10000
I (150) boot: Disabling RNG early entropy source...
Flash size = 4194304
Read from 9000:  [14, 29, 3e, 53, 14, 29, 3e, 53, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff]
Written to 9000: [15, 2b, 41, 57, 15, 2b, 41, 57, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff]
Read from 9000:  [15, 2b, 41, 57, 15, 2b, 41, 57, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff]

I have copied the example code into my own project created with cargo generate esp-rs/esp-template because inside the esp-storage repository, cargo fails to find the xtensa-esp32-non-elf target for some reason.
Maybe you can help me to run the example in the same way as you do?
I get:
> cargo +esp run --release --example demo --features esp32 --target xtensa-esp32-none-elf
    Finished release [optimized] target(s) in 0.08s
     Running `espflash flash --monitor target/xtensa-esp32-none-elf/release/examples/demo`
error: could not execute process `espflash flash --monitor target/xtensa-esp32-none-elf/release/examples/demo` (never executed)

Caused by:
  No such file or directory (os error 2)

> cargo espflash flash --release --example demo --features esp32 --target xtensa-esp32-none-elf
[2023-11-15T11:15:05Z INFO ] Serial port: '/dev/ttyUSB0'
[2023-11-15T11:15:05Z INFO ] Connecting...
[2023-11-15T11:15:05Z INFO ] Using flash stub
error: failed to run `rustc` to learn about target-specific information

Caused by:
  process didn't exit successfully: `/home/nik/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/rustc - --crate-name ___ --print=file-names -C link-arg=-Tlinkall.x --target xtensa-esp32-none-elf --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=split-debuginfo --print=crate-name --print=cfg` (exit status: 1)
  --- stderr
  error: Error loading target specification: Could not find specification for target "xtensa-esp32-none-elf". Run `rustc --print target-list` for a list of built-in targets

Oh, thinking about it… I had to change the demo code at one point, because this doesn't work:
        #[cfg(feature = "esp32")]
        let system = peripherals.DPORT.split();

esp32_hal does not have a DPORT peripheral. I used the SYSTEM peripheral instead. Maybe something changed in esp32_hal here as well?
In general, could you explain the relevance of the time setup code as used in the demo for the Flash access? I don't understand why I need to get a timer group and one time from it, but the timer is never used later on.

Ah maybe you don't have espflash installed? (You have cargo-espflash, see https://crates.io/crates/espflash)
With espflash installed the first command should just work
Or you can try
cargo +esp espflash flash --release --example demo --features esp32 --target xtensa-esp32-none-elf
to run it with the esp-toolchain
In a generated project you don't need +esp since the template creates rust-toolchain.toml for you

The demo uses an older version of ESP-HAL and we renamed DPORT to SYSTEM in our latest release
The whole timer thing in the demo is also because of the older HAL version: now we deactivate the watchdog-timer in HAL before running user-code. Before the user had to deactivate the WDT manually
.... and now I'm wondering if anything changed in esp-hal breaks esp-storage 🤔
So would be great if you could check if it's working in the repository without any changes

Ok doesn't seem like there is anything changed in esp-hal causing any problems here

The example run from a fresh clone of the repository also hangs, both with espflash and cargo-espflash.

Surprisingly, the first read already returns only 0xff bytes:
Flash size = 4194304
Read from 9000:  [ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff]

All 0xff is fine since probably nothing was ever written to that flash page.
Could you post the full output (including messages from the 2nd stage bootloader and the ROM)?

nik@frifot /t/esp-storage (main)> cargo +esp run --release --example demo --features esp32 --target xtensa-esp32-none-elf
    Finished release [optimized] target(s) in 0.03s
     Running `espflash flash --monitor target/xtensa-esp32-none-elf/release/examples/demo`
[2023-11-15T12:37:38Z INFO ] Serial port: '/dev/ttyUSB0'
[2023-11-15T12:37:38Z INFO ] Connecting...
[2023-11-15T12:37:38Z INFO ] Using flash stub
Chip type:         esp32 (revision v1.0)
Crystal frequency: 40MHz
Flash size:        4MB
Features:          WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
MAC address:       c8:c9:a3:c9:a7:ac
App/part. size:    92,384/4,128,768 bytes, 2.24%
[00:00:01] [========================================]      17/17      0x1000                                                      [00:00:00] [========================================]       1/1       0x8000                                                      [00:00:02] [========================================]      26/26      0x10000                                                     [2023-11-15T12:37:43Z INFO ] Flashing has completed!
Commands:
    CTRL+R    Reset chip
    CTRL+C    Exit

ets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:7104
0x3fff0030 - _stack_end_cpu0
    at ??:??
load:0x40078000,len:15576
0x40078000 - ets_delay_us
    at ??:??
load:0x40080400,len:4
0x40080400 - _init_end
    at ??:??
ho 8 tail 4 room 4
load:0x40080404,len:3876
0x40080404 - _init_end
    at ??:??
entry 0x4008064c
0x4008064c - __user_exception
    at ??:??
I (31) boot: ESP-IDF v5.1-beta1-378-gea5e0ff298-dirt 2nd stage bootloader
I (31) boot: compile time Jun  7 2023 07:48:23
I (33) boot: Multicore bootloader
I (37) boot: chip revision: v1.0
I (41) boot.esp32: SPI Speed      : 40MHz
I (46) boot.esp32: SPI Mode       : DIO
I (50) boot.esp32: SPI Flash Size : 4MB
I (55) boot: Enabling RNG early entropy source...
I (60) boot: Partition Table:
I (64) boot: ## Label            Usage          Type ST Offset   Length
I (71) boot:  0 nvs              WiFi data        01 02 00009000 00006000
I (78) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (86) boot:  2 factory          factory app      00 00 00010000 003f0000
I (93) boot: End of partition table
I (97) esp_image: segment 0: paddr=00010020 vaddr=3f400020 size=028e0h ( 10464) map
I (110) esp_image: segment 1: paddr=00012908 vaddr=40080000 size=01fa8h (  8104) load
I (118) esp_image: segment 2: paddr=000148b8 vaddr=00000000 size=0b760h ( 46944)
I (139) esp_image: segment 3: paddr=00020020 vaddr=400d0020 size=06898h ( 26776) map
I (150) boot: Loaded app from partition at offset 0x10000
I (150) boot: Disabling RNG early entropy source...
Flash size = 4194304
Read from 9000:  [ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff, ff]

Good news .... I finally found a board showing that behavior. Bad news: it's a revision 3 (so if I can fix it we don't know if it fixes your problems) and I currently wonder why that is happening only on that board

If it helps, I can give you access to an environment with the board using tmate (SSH).

I traced it to cache_read_disable_rom(0) in https://github.com/esp-rs/esp-storage/blob/f046349fa588343fee3310586722ab119331347b/src/esp32.rs#L44 never returning.

@Natureshadow could you try the branch https://github.com/esp-rs/esp-storage/tree/fix-some-esp32 ?

@Natureshadow could you try the branch https://github.com/esp-rs/esp-storage/tree/fix-some-esp32 ?

Done. Works ☺!

Thanks! Will do some cleanup and then this is hopefully solved

Hi, I can also confirm it works now. Thank you!


Threads stuck in TIMED_WAIT state when calling MultiProducerSequencer::next

Hi, I'm reposting this as an issue, which originated from a comment associated to #164.
I don't believe that there's a bug in the code, but I hope this issue helps in finding specific patterns/configurations to reproduce the aforementioned behavior in the title.
Essentially 1 out of 7 servers in my production environment, started to have troubles with MultiProducerSequencer (disruptor-version 3.3.4). The only difference between the servers is the JVM version, which is @ 1.7.0_60 on the OK ones and 1.7.0_76, everything else is the same (16-core Xeon E7440, 32GB RAM, RHEL 5.5).
I'm using RingBuffer with WorkHandler, essentially in a Many (producer) to One (consumer). The number of consumers is configured to be the same as the number of CPUs.
public EventProcessor(){
//this implements EventFactory, ThreadFactory, ExceptionHandler, WorkHandler
disruptor = new Disruptor<>(this, cap, this, ProducerType.MULTI, new SleepingWaitStrategy());
//
disruptor.setDefaultExceptionHandler(this);
disruptor.handleEventsWithWorkerPool(createWorkers());
}

//creates as many workers as available CPUs
private WorkHandler<EventHolder>[] createWorkers() {
		final WorkHandler[] rv = new WorkHandler[Runtime.getRuntime().availableProcessors()];
		for (int i = 0; i < rv.length; i++) {
			rv[i] = this;
		}

		return rv;
}

//prevent Worker Threads from dying
public void handleEventException(final Throwable ex, final long sequence, final EventHolder event) {
		log.warn(ex.getMessage(), ex);
}

Stack dump shows several ajp Threads (which are used by the load balancer) in a similar state:
ajp-<IP_ADDRESS>-8009-1 tid=355 [WAITING] [DAEMON] (Extracted from heap dump on yjp profiler. JStack shows similar output. YJP does not translate Thread States properly 673=>TIMED_WAITING)
java.util.concurrent.locks.LockSupport.parkNanos(long) LockSupport.java:349
com.lmax.disruptor.MultiProducerSequencer.next(int) MultiProducerSequencer.java:136
com.lmax.disruptor.MultiProducerSequencer.next() MultiProducerSequencer.java:105
com.lmax.disruptor.RingBuffer.publishEvent(EventTranslatorTwoArg, Object, Object) RingBuffer.java:509
---<omitted>---.java:130
---<omitted>---.java:110
---<omitted>---.java:466
---<omitted>---:355
---<omitted>---:173
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse) FilterChainProxy.java:342
---<omitted>---:62
org.springframework.security.web.FilterChainProxy.doFilter(ServletRequest, ServletResponse, FilterChain) FilterChainProxy.java:160
---<omitted>---:37
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(Request, Response) SecurityAssociationValve.java:182
org.terracotta.modules.tomcat.tomcat_5_5.SessionValve55.tcInvoke(Request, Response) SessionValve55.java:100
org.terracotta.modules.tomcat.tomcat_5_5.SessionValve55.invoke(Request, Response) SessionValve55.java:85
org.terracotta.session.ModernTomcatSessionValve.invoke(Request, Response) ModernTomcatSessionValve.java:66
java.lang.Thread.run() Thread.java:745

I collected 2 heap dumps, with a 20min gap between them, and every sequence (cursor,gatingSequenceCache, sequences of gatingSequences) in the MultiProducerSequencer were exactly the same. RingBuffer's entries were all empty (My events hold 2 states, which are set by EventTranslatorTwoArg and set to null after processing).
I don't know if there's some kind of JVM-JIT related issue, but after setting -XX:-UseLoopPredicate on the problematic server, the issue was gone (well, it didn't present itself so far in about a week).

There was a number of concurrency issues with earlier builds of the 1.7 jdk.  I would recommend updating to the latest build revision of 1.7 or even 1.8 if you can.

All of the 22 servers in our production environment have the same problem.
disruptor version: 3.3.4
CentOS release 6.5 (Final)
JDK 1.7.0_80

Do you have a stack trace of the WorkHandler threads.

BTW, from a tuning perspective use the number of available processors is probably not the best option.  This will mean that system will have few CPUs than the number of threads used for processing.  I.e. application threads = publisher threads + worker threads.  If worker threads = # of CPUs then the publisher threads will be contending with the worker threads for CPU resource.

upload my stack trace and
/--------
jstack.log.zip

This is fixed in 3.3.6.


Add conversion tool to take modified frcmod files to SMIRFF FFXML as per #118

Resolves issue #118 by adding to utilities/convert_frosst a script which can take a modified frcmod file as specified in #118 and convert it to SMIRFF FFXML.
Some other minor changes elsewhere made this much easier and were done generally as they will likely be useful in general:

Modified ForceField to add the addParameter function (paralleling getParameter and setParameter) which can introduce a new SMIRKS pattern and associated parameters to a specified force section
Modified ForceField.setParameter to allow it to change the number of parameters associated with a particular SMIRKS, since previously it was impossible to (for example) change a dihedral involving just one k into one involving multiple k values (k1, k1, etc.).

FYI, @bannanc - PRs under the main branch still seem to pass tests, oddly, so there must be something awry in your branch. Very odd. Will investigate when I get time or when it becomes high priority.

@cbayly13 - this adds the functionality you wanted. I'll merge shortly as soon as the tests pass.
You'll find the updated functionality under utilities/convert_frosst; the main conversion functionality is provided by convert_frcmod.py there but the Jupyter notebook you want is parameter_usage.ipynb.


languages: Add clangd_extensions, a plugin to handle clangd LSP extensions

Should be done, sorry for the force push needed it to rebase on master


docs: update the link to the CONTRIBUTING.md file

[x] Have you provided a meaningful PR description?
[ ] Have you added a test, reproducer or referred to an issue with a reproducer?
[ ] Have you tested your changes locally for CPU and GPU devices?
[ ] Have you made sure that new changes do not introduce compiler warnings?
[ ] If this PR is a work in progress, are you filing the PR as a draft?

This PR resolves a broken hyperlink issue in the documentation. The hyperlink should direct users to the CONTRIBUTING file, but currently, it is not functioning correctly. To fix this, the PR modifies the hyperlink to correctly point to the CONTRIBUTING.md file.

Thanks!


Cannot import Prisma JSON scan result file

Hello, I have a question.
I cannot import Prisma JSON scan result file because it return an error as images.
I guess, it cannot read “link” key in JSON file.
How can I do?
I use latest version DefectDojo and have deployed on Kubernetes Cluster.
Thank you.

Can you attach a sample file that is causing the problem?

https://github.com/gun082544/prisma-sample-result/blob/main/prisma-result.json
Here it is.

What scan type are you using to parse this file @gun082544  ?

@coheigea I use Twistlock Image Scan type.

Pr submitted: https://github.com/DefectDojo/django-DefectDojo/pull/7159

@coheigea Thank you. 🙏🏻


Nginx反向代理Google | ilubov

http://blog.ilubov.com/2020/12/06/nginx_google/
Get busy living or get busy dying.

🌚


Added sanitize_np function, which is faster than sanitize_doc

sanitize_doc was a bottleneck for mongo inserting. sanitize_np fixes this problem. I made it a separate function because sanitize_doc also changes tuples to lists in addition to sanitizing numpy, and this feature is useful for some tests.

This does not look like it will catch:
d = {'a': {'b': np.arange(5))}}

Can this be updated to handle a list of arrays? Example:
In [3]: sanitize_np({'a': [numpy.arange(5)]})
Out[3]: {'a': [array([0, 1, 2, 3, 4])]}

We encountered that in the wild at some point and that was our reason for switching to the approach currently used in sanitize_doc. Would be nice to have something that can handle that but is fast, which a round-trip through JSON is not.

Codecov Report

Merging #71 into master will decrease coverage by 0.8%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master      #71      +/-   ##
==========================================
- Coverage   79.89%   79.08%   -0.81%
==========================================
  Files           3        3
  Lines        1104     1119      +15
==========================================
+ Hits          882      885       +3
- Misses        222      234      +12

Impacted Files
Coverage Δ

event_model/__init__.py
83.59% <100%> (-1.95%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2f026ee...6170f54. Read the comment docs.

It would be better to duck-type if we can. Resorting to isinstance creates fragility because it won't work on things are dict-like or list-like. For example, this won't recurse into tuples. See for example https://github.com/NSLS-II/databroker/blob/30e9c707171df5ad14cd58a21382284af7c7db87/databroker/utils.py#L29-L42 where we sniffed out dicts based on the existence of an .items() attribute. A good trick for catching lists/tuples/etc. is to use isinstance(x, collections.abc.Iterable).

I'll add some tests before trying to merge, I'll work on that later.


Improve the end-of-build message

xk6 currently makes assumptions that users are familiar with go build system and understand what $PATH is.
New users often assume that xk6 build process will replace their existing k6 installation. This is not the case, instead, the new build is in a $GOPATH directory, which is usually not part of $PATH.
I suggest the following:

At the end of the xk6 build process we print instructions on where to locate the new build and how to use it
[bonus for extra points] Detect the path of globally installed k6 and give the user the exact command to replace it with the new build (sudo cp /gopath/bin/k6 /usr/local/bin/k6)

Suggestion for the message
***************************************************
* BUILD COMPLETE - FOLLOW THESE STEPS TO CONTINUE *
***************************************************
xk6 has now produced a new k6 binary including "{xk6-redis}" extension.
The new binary is located in {`/home/sniku/go/bin/k6`}.
To use it you need to call it directly `./home/sniku/go/bin/k6 run script.js`
or replace your globally installed k6 with the new one.

Yeah, that would help clarify things. As well as document it better in the README, it's not very clear what happens by default.
Though I wouldn't suggest replacing the globally installed k6 binary. For one it might conflict with package managers if k6 was installed from a package, but it would also cause confusion if they build a second binary and overwrite a previous one that had another extension. This is really up to the user how they want to handle it (e.g. renaming the binary to k6-redis, etc.), but it should be a conscious decision, not a suggestion from us.
At the end of the day though, xk6 is a tool for developers and advanced users. It expects some background knowledge around setting up their Go environment (which is currently also a common issue, see #36 and on the forum), how $PATH works, and how to run a binary from the current working directory. Until we have a more user friendly way of at least installing extensions, some of which we've discussed, there's not much we can do besides improving the documentation and notices like this.


Add tarball

Committed .tgz file that's created after running npm install and npm pack

This repo uses yarn so for generating the Tarball we should run:

yarn install
yarn pack`

Are these steps right?

I always used npm install and npm pack, because I think it better reflects how they are installed on the current Gutenberg (Mobile) CIs.


[V0.31.0] no-bare-urls reports false positive

The new version 0.31.0 of Markdownlint reports a false positive for the following code: demo / reality
False-positive

MD034 / no-bare-urls Bare URL used [Context: "http://foo/com"]

I think it's the HTML detection that's broken in version 0.31.0, as the no-inline-html rule isn't reported for the above code.

As your demo link shows, that syntax is not recognized as HTML and renders as if it were Markdown text. Therefore, I claim it is correct that the "inline HTML" rule does not warn about it and also correct that the embedded URL is flagged as by the "bare URL" rule.


IncrementalLMScorer discards probability of first token

Consider this section of code from IncrementalLMScorer:
## Ignore the probabilities of the first token.
effective_ids = [id[1:] for id in ids]

If I'm understanding this correctly, the class is discarding the probability the model assigns to the first token in every element of a batch. I understand why such logic would make sense in the context of a model that uses a BOS token; but does this mean that this class is unusable for models that do not use a BOS token? It is not at all clear from the docs that this class is only meant to be used with BOS token models.
A BOS token is mentioned in other places in the code- for example, as a Boolean argument to prepare_text- but in such places it's clearly marked as optional, with the default being False. So I'm a little confused by the lack of such optionality in the function above.
Am I understanding this correctly? If so, is there a workaround (besides reduplication of the code)?

Thanks for raising this issue! When LMs don't use a BOS token, it makes no sense to have probabilities of the first token, since logits are computed given some context. This is the default case.
But for cases where LMs do have a BOS token, the first token ends up being the BOS token itself and now it is the BOS token that is assigned 0 probability -- in such cases you could enable the bos_token option by setting it to True.
The compute_stats function doesn't need to include this functionality since all of this is handled by the prepare_text and the prime_text functions, all compute_stats does is handle logits given some input.
Does this make sense?

Thanks for raising this issue! When LMs don't use a BOS token, it makes no sense to have probabilities of the first token, since logits are computed given some context. This is the default case.
But for cases where LMs do have a BOS token, the first token ends up being the BOS token itself and now it is the BOS token that is assigned 0 probability -- in such cases you could enable the bos_token option by setting it to True.
The compute_stats function doesn't need to include this functionality since all of this is handled by the prepare_text and the prime_text functions, all compute_stats does is handle logits given some input.
Does this make sense?

Yes, and thank you for your quick reply. I understand. Would you happen to have any advice for doing a BLiMP type experiment for a model that does not take use a BOS token? If all sentence pairs in the BLiMP corpus had an identical first word, this wouldn't matter, but for some pairs, this is not the case (e.g. matrix_question_npi_licensor_present  and left_branch_island_echo_question). Is the simple LM BLiMP evaluation method intelligible in such cases?

I think you're making an important point here with the difference in first word for LMs without a BOS token! For these cases, I guess the difference in the first token would be more indirectly reflected in the log-probs assigned by the model in the sense that p(apple | an) is likely much much greater than p(apple | a). I am unsure about other solutions...

Thanks for your very helpful responses (and the very helpful library). Yes I did some more thinking about this and I understand your point. I'll close the issue now :)

No worries -- I still think your point holds, fwiw!


Operator and operand code need refactoring

Especially in the operator/operand install code, there is a lot of overlap in what these do. Also, the install functions are doing more than they should have responsibility for.
So, this bug should address:

[X] Dedupe the report code and have it use a template (or a marshaler) #249
[ ] Dedupe the install code.
[ ] Make the install functions just have the responsibility of install, and not of creating reports.

@bcrochet , I guess this issue could be split into two other issues with better explanation and directions for other people to contribute as well. What do you think?

@bcrochet , I guess this issue could be split into two other issues with better explanation and directions for other people to contribute as well. What do you think?

Yes. I will look at what we have gotten to so far and spell out where it needs to go.


[gha] select cluster-test suite based on breaking

Motivation
Resolves #5172
Select whether to run land_blocking or land_blocking_compat experiment suite for cluster-test in LBT based on whether the PR commit message has [breaking]. Having it in the commit message will not only inform LBT which suite to run, but also as an indicator of which commits are breaking without relying on Github PR labels:
git log $(git log origin/testnet --format=format:"%h" -1)..HEAD --pretty=oneline | grep '\[breaking\]''

Configures land_blocking_compat to run compatibility test against the previous commit in master.
NOTE: Now relies on KILL_SWITCH_LAND_BLOCKING_COMPAT Github secret, that has yet to be created.
Have you read the Contributing Guidelines on pull requests?
Yes
Test Plan
Canary in GHA:

with [breaking] flag: https://github.com/libra/libra/runs/892241240?check_suite_focus=true

Related PRs
Depends on https://github.com/libra/libra/pull/5150, which introduces prettier report text for LBT results.

:umbrella: The latest upstream changes (presumably c5328db7293edd6b020acd3b193d910ee67c3877) made this pull request unmergeable. Please resolve the merge conflicts.

Few comments about code:

You want to analyze all commits between origin/master and HEAD - PR can contain more then one commit
I would suggest using smth like pretty=oneline to only search for breaking tag in commit header. I suggest this because this way we can see breaking changes in the #changelog
Ideally can we have some knob that will disable compat test without making commit, similar how we now have killswitch to disable land blocking test?

And a few meta comments on the procedure:

We should make a post announcing this change before we land this diff. Ideally give it at least a day to gather feedback
There might be some confusion about 'breaking' changes, as there are different types of breaking changes. In general, I would say there are server-breaking and client-breaking changes. Current compat test focuses on server-breaking changes, and many people are actually interested in client-breaking changes. Should we have separate labels for those?

Actually, on client vs server breaking I am not so sure. Maybe it will just overcomplicate things and we can just leave it as 'breaking', just highlight that client compatibility check is not perfect at the moment and there will be more work to do in this area

Thanks for the feedback @andll ! few comments:

You want to analyze all commits between origin/master and HEAD - PR can contain more then one commit

I might be wrong, but as far as I'm aware, Bors will squash the PR commits for us in auto, so just checking the previous commit is fine.

Should we have separate labels for those? [client and server breaking changes]

I think we can just leave it as [breaking] for now, and rely on the report text to see what actually broke. I imagine in the future we'll have a separate experiment that will use various clients to test against the cluster, and we can report errors there.

Bor's won't necessarily squash commits at one - I think it depends on some settings, but there are certainly PRs that push more then one commit to master
I think keeping single [breaking] label is ok for now

Ideally can we have some knob that will disable compat test without making commit, similar how we now have killswitch to disable land blocking test?

I think we can just reuse the existing LBT killswitch, unless you mean to still run land_blocking suite and not land_blocking_compat? User can still tag their commit as breaking and try to land their PR again.

Yeah, I meant separate kill switch that will just make all PRs to use land_blocking, regardless of commit message
We have barely tried compat test - I think activating it without ability to quickly disable is a bit dangerous. And disabling entire land blocking test when we discover problem in compat is also far from ideal, as we've seen in the past, disabling land blocking test quickly leads to bad things getting into master

Why do we need to create this dependency based on the [breaking] tag in the commit message?  Shouldn't we run the compat test by default?

@sausagee compat is run by default, but can be avoided by using [breaking] tag if breaking changes must be made. Otherwise, compat test is run by default, and its suite land_blocking_compat is essentially the land_blocking suite but with compat test before running the benchmark

Updated with new DISABLE_FN_UPGRADE=1 from https://github.com/libra/libra/pull/5264

PR looks good
I wonder if there is (easy) way to define repro CMD once, instead of having two pieces of code, once where we execute it, and second time when printing it

@bors-libra r=andll

:pushpin: Commit 7706f89 has been approved by andll

:hourglass: Testing commit 7706f89e42744501548f99f08b1a527b216b0f95 with merge bf32041a1880cbe6a6235d11b3174473c6e3955a...

:sunny: Test successful - checks-actions_land_blocking_test, checks-circle_commit_workflow
Approved by: andll
Pushing bf32041a1880cbe6a6235d11b3174473c6e3955a to master...


Setup failed - Unsupported Device

Version of the custom_component

1.8.5
Describe the bug
Hey there...
During the setup of the integration the connect to CMI failed with an unsupported device error. The log message is pretty clear what's the problem but the reported device 31 is an CoE Endpoint configured on the CMI. Is it not possible to use the CMI and CoE Integration at the same time?
Greetings
rick
Debug log

Logger: custom_components.ta_cmi
Source: custom_components/ta_cmi/config_flow.py:125
Integration: Technische Alternative C.M.I. (documentation, issues)
First occurred: 15:59:19 (1 occurrences)
Last logged: 15:59:19

Error while communicating with a device (31): Device not supported

2023-08-15 16:27:42.461 DEBUG (MainThread) [custom_components.ta_cmi] Sleep mode for 75 seconds to prevent rate limiting
2023-08-15 16:28:57.463 DEBUG (MainThread) [custom_components.ta_cmi] Try to fetch device type: 10
2023-08-15 16:28:57.875 DEBUG (MainThread) [custom_components.ta_cmi] Sleep mode for 75 seconds to prevent rate limiting
2023-08-15 16:30:12.877 DEBUG (MainThread) [custom_components.ta_cmi] Try to fetch available device channels: 10
2023-08-15 16:30:13.919 DEBUG (MainThread) [custom_components.ta_cmi] Sleep mode for 75 seconds to prevent rate limiting
2023-08-15 16:31:28.921 DEBUG (MainThread) [custom_components.ta_cmi] Try to fetch device type: 31
2023-08-15 16:31:28.932 WARNING (MainThread) [custom_components.ta_cmi] Error while communicating with a device (31): Device not supported
2023-08-15 16:31:28.943 DEBUG (MainThread) [custom_components.ta_cmi] Sleep mode for 75 seconds to prevent rate limiting
2023-08-15 16:32:43.945 DEBUG (MainThread) [custom_components.ta_cmi] Try to fetch device type: 50
2023-08-15 16:32:44.190 DEBUG (MainThread) [custom_components.ta_cmi] Sleep mode for 75 seconds to prevent rate limiting
2023-08-15 16:33:59.191 DEBUG (MainThread) [custom_components.ta_cmi] Try to fetch available device channels: 50
2023-08-15 16:34:14.393 ERROR (MainThread) [frontend.js.latest.202308020] :0:0 Script error.

Can you finish setting up the integration and the other devices? If so, you can ignore the error message, since the device (31) is only communicated with in the setup process and the error does not occur afterwards.
The message is only a hint why some devices are not shown on the list for selection. In your case, it is the CoE node that is not supported by the API.

Well... i couldn't finish setting up the integration as I wrote this issue.
Now I've tried a couple more time and made it to next dialog where I can choose the devices.
The dialoag shows an "communication error with one device".
I selected then my controller and added it but had no entities available.  Proabably because I deselected the checkbox for all channels.
On the next try I could add my controller with entites but I am missing the valve output which are two outputs paired.
Can I edit the channels after adding a device?

Unfortunately, not yet. I have not yet found a good way to integrate it.


Problems using swing with PowerMockRunner

When trying to instantiate a swing component in a test class run with PowerMockRunner, I get a VerifyError in swing.UIManager, see the trace in the attached file. I use powermock-mockito-junit-1.6.3.
I'm running Java 8 on Ubuntu 15.10 and I use Eclipse to write and run my code.
To reproduce the problem, this class can be used:
import javax.swing.JFrame;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.powermock.modules.junit4.PowerMockRunner;

@RunWith(PowerMockRunner.class)
public class SwingClassloaderTest {
	@Test
	public void test() {
		JFrame frame = new JFrame();
	}
}

VerifyErrorTrace.txt

I'm getting the exact same problem. Have you figured it out by any chance?

Unfortunately not. :( I think this is a bug in Powermockito.
One workaround is to defer loading of the swing classes (using the annotation @PowermockIgnore), but then you cannot do PrepareForTest on swing components.

I didn't have experience with Swing, but issue looks the same as with another framework which either uses custom classloader or work only with system class loader or work in another thread than test classes.
Root clause: PowerMock loads classes by using custom class loader which modified classes during loading, but Swing doesn't know nothing about this class loader. I don't know if it possible to force Swing use custom class loader.
From PowerMock side it's hard to fix the issue, because PowerMock should be aware that a test is run with some framework which requires special initialisation. As in this case.

@kohanian There is a workaround. (Applying this workaround, I get a LinkageError instead, but it seems to be a known issue.)


fix: fix popover content is overflow in PropsTable

While researching the documentation of rainbowkit, I identified a small issue related to the UI here.
The problem originates from the data cell td element that was a style white-space: nowrap. The popover children subsequently inherit this style, causing a UI glitch. So, I have reset the white-space within the popover content.
Please find the before and after screenshots below for a better understanding of the changes:
Before:

After:

@monodyle Thank you!


Add LessPass

Signed-off-by: Shivam Kumar<EMAIL_ADDRESS>

@ShivamKumar2002 , thanks for your PR!
Is it a company?

@ShivamKumar2002 , thanks for your PR!
Is it a company?

Not sure..
https://www.crunchbase.com/organization/keepass
https://www.datanyze.com/companies/keepass/354481602
Here its showing as company


Unit tests addition

Moving towards #101

Current coverage is 73.41%

Merging #104 into master will increase coverage by +3.87% as of ba445fa

@@            master    #104   diff @@
======================================
  Files           50      50
  Stmts         1008    1008
  Branches       171     170     -1
  Methods          0       0
======================================
+ Hit            701     740    +39
+ Partial         43      42     -1
+ Missed         264     226    -38

Review entire Coverage Diff as of ba445fa

Powered by Codecov. Updated on successful CI builds.


Large TAGS file

Hi
I had difficulty in using a fairly large tag file - my tag file size is 504M and it would not load properly - from Linux Kernel Source Browsing

Thanks, @prama01720. I'll take a look at this.

@prama01720 this issue should be fixed in the next release.

Thank you. In case you need me to test anything please let me know. I am willing to help.
Regards
Rama
On Wednesday, November 14, 2018, 1:09:27 PM EST, Chris Wheeldon<EMAIL_ADDRESS>wrote:

@prama01720 this issue should be fixed in the next release.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Thank you. In case you need me to test anything please let me know. I am willing to help. Regards

@prama01720 , please give version 0.0.7 (the latest published version) a try and see if that now works for you. Thanks.


Long URLs get truncated by wget

Wget truncates long URLs when storing them as HTML files on disk.
For example, this URL is 286 characters:
https://www.consumerfinance.gov/ask-cfpb/i-found-the-car-of-my-dreams-but-the-dealer-says-that-i-have-to-have-a-down-payment-when-i-said-i-didnt-have-the-money-the-dealer-said-that-if-i-add-a-gps-and-a-stereo-he-will-be-able-to-get-me-a-loan-for-the-full-amount-what-should-i-do-en-835/
When converted to a file named index.html under the www.consumerfinance.gov folder, it would be 288 characters:
www.consumerfinance.gov/ask-cfpb/i-found-the-car-of-my-dreams-but-the-dealer-says-that-i-have-to-have-a-down-payment-when-i-said-i-didnt-have-the-money-the-dealer-said-that-if-i-add-a-gps-and-a-stereo-he-will-be-able-to-get-me-a-loan-for-the-full-amount-what-should-i-do-en-835/index.html
Wget truncates this to 236 characters, as seen in the log:
The name is too long, 288 chars total.
Trying to shorten...
New name is www.consumerfinance.gov/ask-cfpb/i-found-the-car-of-my-dreams-but-the-dealer-says-that-i-have-to-have-a-down-payment-when-i-said-i-didnt-have-the-money-the-dealer-said-that-if-i-add-a-gps-and-a-stereo-he-will-be-able-to-get-me-a-loan-fo.

(I believe this is done because the filename can be at max 255 characters, and wget has a 19-character "chomp buffer" that it reserves for appending things like .tmp, /index.html, etc.)
Note that this new filename doesn't have an extension. Wget then tries to download to this location:
--2020-10-30 18:57:07--  https://www.consumerfinance.gov/ask-cfpb/i-found-the-car-of-my-dreams-but-the-dealer-says-that-i-have-to-have-a-down-payment-when-i-said-i-didnt-have-the-money-the-dealer-said-that-if-i-add-a-gps-and-a-stereo-he-will-be-able-to-get-me-a-loan-for-the-full-amount-what-should-i-do-en-835/
...
Saving to ‘www.consumerfinance.gov/ask-cfpb/i-found-the-car-of-my-dreams-but-the-dealer-says-that-i-have-to-have-a-down-payment-when-i-said-i-didnt-have-the-money-the-dealer-said-that-if-i-add-a-gps-and-a-stereo-he-will-be-able-to-get-me-a-loan-fo.tmp’
...
Removing www.consumerfinance.gov/ask-cfpb/i-found-the-car-of-my-dreams-but-the-dealer-says-that-i-have-to-have-a-down-payment-when-i-said-i-didnt-have-the-money-the-dealer-said-that-if-i-add-a-gps-and-a-stereo-he-will-be-able-to-get-me-a-loan-fo.tmp since it should be rejected.

But it then removes the downloaded file because it doesn't end in .html! (I believe that wget adds but then properly ignores the .tmp suffix it  adds when downloading files.)
Current behavior
This truncating behavior causes a few problems:

Because we only --accept html, files that get truncated to end with some other extension (or get no extension at all) will be deleted, and won't be tracked in this repository. (See also https://github.com/cfpb/crawl-cfgov/issues/9#issuecomment-719752504; this probably needs to be fixed by instead specifically using --reject on certain non-HTML files like .pdf,.jpg, etc.) (FIXED by #15)
Any downstream code that depends on these files adding in .html won't work correctly. For example, generate_summary.sh currently deliberately diffs only HTML. Additionally, a file could (at least in theory) get truncated to end in an extension specified in our .gitignore!
The diffs and output files in this repo become harder to use if some files don't end in .html. They become harder to edit locally and generally just less useable for downstream applications.

Expected behavior
It would be better if we could somehow ensure that all files always get saved consistently as .html, or, at least makes  it easier to track when this happens. As far as I can tell we can't do this with wget itself.
One idea would be to write a script that parses our wget.log file to generate a list of URLs and their truncated filenames. We could then have some other script that "corrects" those filenames.

Problem 1 above (non-HTML files getting rejected) was resolved by #15.
Problem 2 above (downstream code relying on .html) will be resolved by #19.
Problem 3 above (complexity of having truncated filenames) is still a potential issue. Let's wait to see what kinds of issues this creates before we decide if we need to fix this or what the best solution might be.


Error when load on new version wolfram engine (or mathematica), version >= 12.1

I tried on local computer
12.1.0 for Linux x86 (64-bit) (April 14, 2020)

It load, with some error.
I then tried it on wolfram cloud.
13.0.0 for Linux x86 (64-bit) (November 22, 2021)

LoadRules
::inv
:Couldnotloadfileorsection:{/wolframcloud/userfiles/d50/d5055238-6028-4d1a-a70a-b024a624b83c/Base/Paclets/Repository/Rubi-<IP_ADDRESS>/IntegrationRules/9Miscellaneous\9.1Integrandsimplificationrules.m}

LoadRules
::inv
:Couldnotloadfileorsection:{/wolframcloud/userfiles/d50/d5055238-6028-4d1a-a70a-b024a624b83c/Base/Paclets/Repository/Rubi-<IP_ADDRESS>/IntegrationRules/1Algebraicfunctions\1.1Binomialproducts\1.1.1Linear\<IP_ADDRESS>(a+bx)^m.m}

LoadRules
::inv
:Couldnotloadfileorsection:{/wolframcloud/userfiles/d50/d5055238-6028-4d1a-a70a-b024a624b83c/Base/Paclets/Repository/Rubi-<IP_ADDRESS>/IntegrationRules/1Algebraicfunctions\1.1Binomialproducts\1.1.1Linear\<IP_ADDRESS>(a+bx)^m(c+dx)^n.m}

General
:FurtheroutputofLoadRules::invwillbesuppressedduringthiscalculation.

If type
??Int

return normal result.
local

wolfram cloud

but it can't calculate.

Your error message looks like there is a mix of / and \ in the filename which might be the reason. Loading the integration rules is a crucial part to make Rubi work.

At first, I install it by https://resources.wolframcloud.com/FunctionRepository/resources/GitHubInstall/
https://www.wolframcloud.com/obj/d8ae15ff-99e0-4b52-8938-0617c4f595a1
then I use the way in manual.
it works.
https://www.wolframcloud.com/obj/c7d25428-12f6-4bd2-a5ff-dcd8f607d2d3
So there may be some problem with GitHubInstall, Rubi itself has no problem.


adc/adc_nrf52: Handle ADC calibration done event.

This PR  fixes assert() when ADC offset calibration is done with nrfx_saadc_calibrate_offset()
This PR does not provide HAL API to trigger ADC calibration (maybe it should be simple adc_offset_calibrate() ?),
however last patch adds support to send ADC_EVENT_CALIBRATED to the user when calibration is completed.
https://github.com/apache/mynewt-core/commit/ec294361aa11bb3ea425f13f45b9404b7a7cf784
For now I just need first two patches, however open to hear comments on API for this.

@andrzej-kaczmarek thx - fixed


OSError: [Errno 2] No such file or directory on Mac ElCapitan

Hi, got this when trying to call generate_barcode...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/treepoem/__init__.py", line 123, in generate_barcode
    bbox_lines = _get_bbox(code)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/treepoem/__init__.py", line 84, in _get_bbox
    stderr=subprocess.PIPE,
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

This is probably because you don't have Ghostscript installed: https://github.com/YPlan/treepoem/blob/master/treepoem/init.py#L79
Sorry the README doesn't document any install process

N.B. on mac it's brew install ghostscript

Documenting in #26

thanks 👍


Test

@jeroen22319 Please sign the Contributor License Agreement!
Click here to manually synchronize the status of this Pull Request.
See the FAQ for frequently asked questions.


Farewarning, I got a crazy bill from Heroku today.

No idea what the problem is, but I am shutting down my app for the time being.
heroku bill

I feel a bit better.
It seems to be a Heroku billing issue.

This freaked me out too, especially since the crazy billing started right after I changed the scheduler a bit added a worker dyno in PR https://github.com/swanson/stringer/pull/412

Thanks for reporting, will close unless we determine the billing is unrelated to that issue :)

This was indeed caused by the the "scheduler" issue.
After reviewing Heroku terms, it appears that their scheduler addon can actually result in charges even on free plans.
They consider the scheduler to be a separate dyno which is billed regularly, but is refunded up to $34.50 (one month of continuous operation). So to have scheduler addons, a credit card is needed and if jobs consume too much time, this will also lead to cost overruns.
Perhaps it would make sense to use a worker instead of a scheduler?
while true; do bundle exec rake fetch_feeds; bundle exec rake cleanup_old_stories sleep 9000; done
This would require no configuration, avoid using the potentially paid addon and also reduce the load on Heroku servers since it won't run while a dyno is sleeping.


Fix failure to parse UTF8 files with BOM

The setup.py for azure-cosmos contains a BOM which was causing our parser to bail out. Unfortunately some dependency inconsistencies were introduced in azure-cosmos which we never caught because of this bug. This fixes the bug but builds will fail due to the broken dependencies in azure-cosmos.

Can one of the admins verify this patch?

Filed #6161 to track fixing the dependencies for azure-cosmos


fix(protoc): Run protoc container as invoking user

style(proto): Add newlines in docker command
This will make it easier to see what changed when adding/removing arguments to the command.

refactor(proto): Use mount instead of volume command
The mount command is newer and recommended by Docker. It also allows us to mount a volume readonly, which we want to do for the proto directory (which is input only).

refactor(protoc): Use better directories in container
It's customary for mounted filesystems to be in /mnt; let's mount our files there instead of directly in the root of the filesystem.
Rather than have a top-level staging directory, make this directory in /tmp. Also create it as part of the genproto script so it's automatically owned by whichever user is running the script.

fix(protoc): Fix file permissions in kleat-protobuilder
This commit updates the intermediate containers used to fetch protoc and its dependencies to no longer run as root.
The first change is to update the 'curl' container to run as a non-root user 'curl' to set the work directory to this user's home directory.
After downloading and unzipping the desired files, we run chmod to set the desired permissions, which in many cases are not set to what we want in the tar/zip archive. We want 755 for directories and executable files and 644 for non-executable files, so that only the owner can modify but anyone can use the files when running protoc.
When copying the files to the kleat-protobuilder container, explicitly have them owned by root (leaving the permissions as what we set them to after downloading). This matches the way things are normally installed on linux, where root owns the files and is the only user with permission to write, but any user can read/execute the files.
In the case of genproto, we'll explicitly follow up with a chmod to ensure it has mode 755 as we can't necessarily control what the user has on their system.
As of this commit, we've fixed the permissions so that it is now possible to run the kleat-protobuilder as a non-root user by passing the -u flag to docker, but we're not actually doing that yet, which will follow in an upcoming commit.

fix(protoc): Run protoc container as invoking user
This commit adds the -u flag to the docker command running the protocol buffer compilation, so that the container runs with the same uid/gid as the invoking user. This will cause any output files to have this uid/gid (instead of being owned by root as before).
I've also created a user 'protoc' in the container and set the user to that user. This is really just a fallback so we're falling back to a non-root user; this will always be overridden by the --user flag that we've added to the docker run command.

It looks like MacOs does some translation, so that even when the container is running as root, the files it writes to mounted volumes are still owned by the user who invoked docker, which is why I didn't notice this before.  I ran make proto from my Linux machine recently and noticed that all the files were owned by root which was annoying as then I couldn't overwrite them without sudo.


Create synthetic wiki index version of sitemaps for wiki depth debugging purposes.

As a search console user, I would like to know what pages underneath one particular wiki are indexed.

Done by https://github.com/nelsonjchen/github-wiki-see-rs/commit/789ec9cbdab1d329b1a1d3245029f0649deceb6e


Router does not re-renders class based Components but does for pure functions

I am not sure if this is a bug in react-router or my code. I am a beginner with React-Router so please excuse my lack of experience.

Version
4.1.1
Test Case
https://codesandbox.io/s/github/vinitkumar/github/tree/master/
Steps to reproduce
On clicking the links on the left side, the render only happens once for Gist Component. After that
the route changes, the props update, but the re-render of Gist component never happens.
Expected Behavior
The Gist component should render fresh every time the props change.
Actual Behavior
The Gist component does not render again after the first render.

React is re-using the previous Gist component instance, but isn't picking up the prop change. Easiest thing to do is to add a key: <Gist key={match.params.gistId} ...

Awesome. Thanks for the help :) 👍


FDRS OLED features

Add OLED 'pages' and blank OLED screen
Adding "paging" feature to OLED such that different sets of data can be shown.  First page will be standard debug data.  Rest of pages will be defined later.  #define in config file states how long each page will stay on screen (default 30 seconds).  When any debug information is added screen immediately switches to that debug screen until there is no action for a specified amount of time (default 30 seconds).

Future enhancements could include but not limited to adding additional "pages" and use library that does not have to be modified.

Quick question: in VSCode, I like to use the autoformatter to make my messy code look better. It currently changes any occurrence like this:
void foo(){
  bar();
}

to
void foo()
{
  bar();
}

I prefer the former, but what can I change in VScode to make it do it right?

Agreed, I prefer the more condensed code as far as brackets go.  I've never used the autoformatter but I'll spend some time researching and let you know what I find.

Looks like you might need an add-on called prettier:
https://www.reddit.com/r/vscode/comments/115h9d1/how_to_change_curly_brackets_behavior_on_file_save/
https://prettier.io/docs/en/rationale.html#empty-lines

This PR should be complete.  I don't plan to add any other pages for awhile.   I wanted to get the "blank page" stuff in there primarily.

Merging this and OTA will be my first task once I get back to work on things. I'm a little behind on things and splitting time with my day-job now.

No worries, take your time. I know you're busy.

Not sure if I said this elsewhere, but this is nifty and I appreciate it!


Refactor popup css positioning

Todo

[x] Refactor instances of absolute positioning in the popup layout
[x] Refactor instances of hard-coded width/height values
[x] Remove conflicting or redundant CSS layout properties
[x] Add back the correct padding/margin/spacing configurations
[x] Update popup.html and popup.js as needed

Motivation
Many components in the extension UI previously used absolute positioning (and fixed width/height). This means additions to the UI in the future will be difficult and more manual. I imagine one would need to use a "trial-and-error" approach on the pixel positioning of the new component.
Summary of changes

Flex positioning was used instead of absolute positioning.
Some class properties were added/removed hence popup.html needed to be changed (more below).
There were many CSS layout properties that were or became redundant and these were deleted. This is why more lines are deleted than added in the PR.
content has a min-height as the extension would look short/crammed if there weren't many social media platforms in the toggle list.
The sites/keywords and sites-content and keywords-content classes have been grouped and configured together in the CSS since they essentially have the same styling.
The top class has been removed since this is redundant.
The content for each keyword/site panel is wrapped in the new classes sites-content and keywords-content to allow for better layout control. The extra navy blue global section is part of the sites class but not the sites-content class and so, the padding does not apply.
Previously, showing/hiding components set the visibility CSS property. This worked because the UI components overlapped each other in the layout due to their absolute positioning. This was changed to display (flex or none) so the UI components don't take up space/jiggle the layout when they are hidden. As a result, changes to popup.js were made.
Additionally, a reference to the list of tags (tag-list) was added to popup.js to control the show/hide logic of it. Previously, it just was covered by the textArea.
The styling for the Save/Cancel buttons needed to match the Edit Allowed/Blocked button to avoid the layout resizing whenever toggling between the edit and view modes for the keywords.
Slight margin/padding and alignment fixes (see screenshots below).

Screenshots

Before Refactoring
After Refactoring

Attached GitHub issue
Closes #96
Checklist
Documentation
Local Build

[x] Extension has been tested to build correctly cd extension && yarn && yarn build
[x] Extension test suite has been run locally cd extension && yarn && yarn test

GitHub

[x] Target branch has been set correctly
[x] PR has been rebased onto target branch
[x] PR has been assigned an owner
[x] Author has performed a self-review of the code
[x] Removed the WIP message from the top of this PR

Codecov Report

Merging #98 (3da140d) into Release-2.0 (954b88b) will decrease coverage by 0.10%.
The diff coverage is 50.00%.

:exclamation: Current head 3da140d differs from pull request most recent head e06b3eb. Consider uploading reports for the commit e06b3eb to get more accurate results

@@               Coverage Diff               @@
##           Release-2.0      #98      +/-   ##
===============================================
- Coverage        69.35%   69.25%   -0.11%
===============================================
  Files               12       12
  Lines              385      387       +2
  Branches            56       56
===============================================
+ Hits               267      268       +1
- Misses              85       86       +1
  Partials            33       33

Impacted Files
Coverage Δ

extension/public/popup.js
55.75% <50.00%> (-0.11%)
:arrow_down:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Exported method TryRuntime_on_runtime_upgrade is not found

Is there an existing issue?

[X] I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

[X] This is not a support question.

 Your test chain spec is loaded from a file and this chain spec doesn't contain the wasm with the required runtime api.


Use compatible docker images in integration test

Currently, tests are failing because incompatible versions of docker images of celestia-app and celestia-node are used.
celestiaorg/ephemeral-cluster already use correct version of images, so the task should be relatively easy.

node v0.5.0-rc5 is compatible with core/app v0.10.0-rc1. Those versions should be used.


Update to version 2.3

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

Cheers for the merge! :+1:


No helper method on class

Hi,
I have a class that has hydra_attributes. I can add attributes on it, select object with these attributes. Everything is working fin except that my class doesnt implement the helpers method (https://github.com/kostyantyn/hydra_attribute/wiki/Helper-methods).
My setup is pretty basic: installed the gem, generate the migration and rake db:migrate it. added the include HydraAttribute::ActiveRecord on the class...
I am using 4.1 branch.
Thank you

Up, can't use hydra_attribute_names_by_backend_type...
PaiementTerm.hydra_attributes #works
PaiementTerm.hydra_attribute_names_by_backend_type #fail
undefined method `hydra_attribute_names_by_backend_type' for #<Class:0x007fb80060c588>


crashing

Unhandled exception "EXCEPTION_ACCESS_VIOLATION" at 0x7FF710C838F0 Fallout4.exe+1B938F0	mov eax, [rcx+0x0C] |  BSResourceNiBinaryStream::BSResourceNiBinaryStream(BSTSmartPointerBSResource::Stream,BSTSmartPointerIntrusiveRefCount&,bool,bool)_1B938F0)
Half the mods i install cause this idk what it means

We'd need the full crashlog to do anything. Please post that next time.


feat(tab): add uniqueName to MatTab (#13921)

Hi guys,
My first PR on Angular/Material2. Please have a look review this and let me know what have I missed.
Implement feature for issue #13921

Closing since the initial need for this can be addressed using harnesses.


Monster of the Week Spellslinger moves broken

The spellslinger moves no longer show up, and instead appear like this:

It appears to have been broken by 0225fd92f889f1f00bac623b070978a89b50a171.

Can confirm. I have fixed it in my fork but will wait until next week before making a pull request as I'm also adding the playbooks from Tome of Mysteries.

@Osspial was this fixed by Anistuff's update? If so, you could close this issue.


SPFX version 1.19.0 - @microsoft/sp-build-web": "1.20.1 not open pdf.js

Target SharePoint environment
SharePoint Online
What SharePoint development model, framework, SDK or API is this about?
💥 SharePoint Framework
Developer environment
Windows
What browser(s) / client(s) have you tested

[ ] 💥 Internet Explorer
[ ] 💥 Microsoft Edge
[X] 💥 Google Chrome
[ ] 💥 FireFox
[ ] 💥 Safari
[ ] mobile (iOS/iPadOS)
[ ] mobile (Android)
[ ] not applicable
[ ] other (enter in the "Additional environment details" area below)

Additional environment details

browser version - latest
SPFx version - 1.19.0
Node.js version - v18.19.0

Describe the bug / error
I wanted upgrade my spfx project to version 1.19.0 cause i am using library PDF.js, which need webpack 5 and higher or additional loader (until now I have used babel-loader).
Few months ago when was beta version of 1.19.0 release I tried it and everything worked as it should.
But I was waiting for the official version.
Unfortunately, I tried to upgrade it today after upgrading this dependency:
"@microsoft/sp-build-web": "1.18.1", this "@microsoft/sp-build-web": "1.20.1"
my pdf.js library stops working with the error:
Unexpected token 'catch' (at pdf.mjs:19862:1)
 at ./node_modules/pdfjs-dist/build/pdf.mjs (bundle.js:1035:1)
 at __nested_webpack_require_2262005_
..
/* harmony import */ var pdfjs_dist__WEBPACK_IMPORTED_MODULE_2__ = __webpack_require__(/*! pdfjs-dist */ "./node_modules/pdfjs-dist/build/pdf.mjs");
..
catch(e) { __webpack_async_result__(e); } }, 1);

when I upgrade project and only this library downgrade everything works, but i am not sure if its best practices..

Problem is when i downgrade i cant use production build cause webpack5 not include and then i got back error:

I still don't see why the dependencies are in version 1.19.0
and devDependencies 1.20.1
Steps to reproduce

Create SPFX webpart
Create Project with pdf.js Lib  and by webpack bundle  or maybe just install pdfjs-dist
Run

Expected behavior
Open the pdf document:

Here is a sample application of how pdf.js and webpack could be used. After npm run build i will get dev.js and prod.js.
After import i have got problem [Object Object].
But I found this in the second issue: Extremely hack-y way to debug this problem
After I finally got some mean error message:
Module not found: Error: Can't resolve 'http' - BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.  ...
Module not found: Error: Can't resolve 'canvas' ...
Module not found: Error: Can't resolve 'fs' ...
Module not found: Error: Can't resolve 'path2d'...
Module not found: Error: Can't resolve 'https' ...
Module not found: Error: Can't resolve 'url' ...

Is there a way to setup gulpfile.js? Or is it an error when exporting bundle.js using webpack?

Or simply debug it in a proper way without manipulation of the build chain with:
https://github.com/SharePoint/sp-dev-docs/issues/9675#issuecomment-2151825299
and
https://github.com/SharePoint/sp-dev-docs/issues/9675#issuecomment-2151867681

Or simply debug it in a proper way without manipulation of the build chain with: #9675 (comment) and #9675 (comment)

I probably don't understand..
gulp bundle  doesn't work - only returned error [Object Object]
that's why I used the overwritten line in :
<EMAIL_ADDRESS>and then I found the errors listed above.
Mainly problem is:
webpack < 5 used to include polyfills for node.js core modules by default.\nThis is no longer the case.
Verify if you need this module and configure a polyfill for it.\n\nIf you want to include a polyfill,
so I am asking how deal with it?
I found that I should put something like this in webpack.js:
module.exports = {
...
 plugins: [
 new NodePolyfillPlugin(),
 ],
 resolve: {
 fallback: {
 "http": require.resolve("stream-http"),
 "https": require.resolve("https-browserify"),
 "url": require.resolve("url/"),
 "fs": false,
 "path2d": false,
 "canvas": require.resolve("canvas"),
 }
 }
};

Do I really need to install all the packages manually?

gulp bundlestill works and because in a dev build it do not really matter or rely on webpack. It prepares in additional file to have to embed the webpart on a page.
Rename the SPPKG file to a .zip and extract on a working project and you won't find any javascript at all in there.
Hence you can still deploy and install the web part and server everything from localhost.
In fact the webpack is only used to load your solution and do nothing during the code compilation.
So deploy a dev build without the --ship - gulp serve from localhost and embedd the web part on the page.
Magically you will see all the npm packages in the browser that are missing for you solution.
More on that on: https://n8d.at/debug-and-fix-webpack-object-object-error-in-sharepoint-web-parts

Even if I use your instructions, as I already announced it's not showing an error.
I did the same as written, the component was loaded, only this will be written at the beginning:
[13:40:37] Error - [webpack] 'dist':
[object Object]
[object Object]
[object Object]
[object Object]
[object Object]
[object Object]
No problem is displayed, neither in the terminal nor in the web part - here the component is loaded as it should!
The problem is that when I
gulp clean
gulp bundle --ship

I get an error here
Error - [webpack] 'dist':
[object Object]

and I'll conclude
gulp package-solution --ship
so I get an error:
Error - [package-solution] No manifests were found in
but if I grant:
gulp bundle
gulp package-solution --ship

so everything seems to have passed, but of course the package was only for the local version
I keep thinking that if I change the line
 this.logError('${outputDir}':+ os_1.EOL + statsResult.errors.map(x => JSON.stringify(x, null, 2)).join(os_1.EOL) + os_1.EOL);
and call:
gulp bundle > output.txt
I get the whole debug + error:
"message": "Module not found: Error: Can't resolve 'https' in 'C:...\node_modules\dist'\n\nBREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.\nThis is no longer the case. Verify if you need this module and configure a polyfill for it.\n\nIf you want to include a polyfill, you need to:\n\t- add a fallback 'resolve. fallback: { "https": require.resolve("https-browserify") }'\n\t- install 'https-browserify'\nIf you don't want to include a polyfill, you can use an empty module like this:\n\tresolve.fallback: { "https": false }", [39m
+5 more errors show above.

Done't use --ship on package-solution this will give you a production build. You can deploy any web part to any site and load the webpart from localhost when you don't use the --ship options.
This option is exclusively for production build, not when you develope a web part.
So the only thing to do is:
gulp bundle
gulp package-solution

Deploy the web part regularly and you will find the issue in your browser like I wrote in my blog post.
In your case to fix your issue:
npm install --save-dev https-browserify

As I wrote, there were more problems and I asked if there was anything better than installing everything manually.
Anyway, what helped me (install all missing packages manually):
npm install --save-dev node-polyfill-webpack-plugin https-browserify stream-http url canvas path2d-polyfill
edit gulpfile.js:
const build = require('@microsoft/sp-build-web');
const webpackConfig = require('./webpack.extend.js');

webpack.extend.js:

const NodePolyfillPlugin = require('node-polyfill-webpack-plugin');
const path = require('path'); // Add this line

module.exports = (config) => {
 config.plugins.push(new NodePolyfillPlugin());

 config.resolve = {
 ...config.resolve,
 fallback: {
 ...config.resolve.fallback,
 "http": require.resolve("stream-http"),
 "https": require.resolve("https-browserify"),
 "url": require.resolve("url/"),
 "fs": false,
 "path2d": path.resolve(__dirname, 'node_modules/path2d-polyfill/dist/path2d-polyfill.js'),
 "canvas": require.resolve("canvas")
 }
 };

 return configuration;
};

Actually the production version works beautifully, the only problem/warning
Warning - [webpack] 'dist':
2:63115-63137: Critical dependency: the request of a dependency is an expression
The problem is somewhere in require :smile: ...
But I still don't know if I'm doing everything right.
Here is the my code for SPFx webpart : spfx-webpack5
Otherwise, I will be glad for any advice/ideas and questions.


Making Book Corpus

Building the same corpus with original paper. Please share your tips to preprocess and download the file. It would be great to share preprocessed data using dropbox or google drive etc.

#32

The original paper (BERT) use "the concatenation of BooksCorpus (800M words) (Zhu et al., 2015) and English Wikipedia (2,500M words)." what do you mean "Movie Corpus"?

@mapingshuo Sorry It's my fault. haha I just made that title in 5seconds :) thank you!! 👍

That's okay, I am looking for a valid Book Corpus too.


Match all types in a map

The rule that try to match all types from the expected result does not work if the actual result has more fields:
Matching rule:
"$.body.*": {"match": "type"}
Expected result:
{
  "userId": "anUserID",
  "email: "anEmail"
}

Actual result:
{
  "userId": "user-1234",
  "age": 40,
  "email<EMAIL_ADDRESS>}

I write a test for the module "pact-specification-test" using v2 of the pact specification (to include in the folder pact-specification-test/src/main/resources/v2/response/body).
match all types in a map.json.zip

Sorry, I misunderstand the use of wildcard in a map.
My case is conflicting with the oficial test case https://github.com/pact-foundation/pact-specification/blob/version-2/testcases/response/body/additional property with type matcher that does not match.json


Docker link settings

fixes issues #565 and #597

@brittainhard I'm running through the manual testing guide with this, but can you code review it?


[Cloud Masthead]: Megamenu height is not displaying correctly within the Drupal integration

Detailed description

While testing the new <dds-cloud-masthead-container> Web Components in Drupal, I noticed that the megamenu height is inheriting min-height: 20rem; instead of the height shown within the Storybook preview.

Steps to reproduce the issue

Visit this Drupal preview page (VPN requires) - https://wc-masthead-jfbm8ry4yxvz0jplleccg7urlffpem5u.w3-origin-007522.intranet.ibm.com/cloud/watson-assistant
Click on the "Products" mega menu item
The height is not the same as what is being shown here - (see - http://ibmdotcom-web-components-experimental.mybluemix.net/?path=/story/components-cloud-masthead--default)

Additional information

HC Engineer @proeung will be working on this ticket.
JIRA ticket: https://jsw.ibm.com/browse/HC-2087

@jeffchew I'm currently working on this ticket and will have a PR for review soon.


Delay first setActive call until event handlers are set up

UpdateFrame is certainly called after the first ProcessEvents https://github.com/ppy/opentk/blob/21754ca665f1b6595c2ef0a02bd4e8ee6fe8dc99/src/OpenTK/GameWindow.cs#L368-L374.

Closes #578

Fixes game not always being active on startup, even when window is.

Hard to say whether this resolved the issue since it happens very rarely, but looks promising.


Can you please add a command in the README for macOS?

Can you please add a command to run the build process on macOS via docker or podman?

Running <podman-or-docker> run -it --privileged nixos/nix bash will drop you into a Linux shell in which you can run the command given in the README (after cloning this repo, of course). After the build is done, you can copy the ISO back to the host using <podman-or-docker> cp. Note that you might need to enable the flakes and nix-command experimental features.
PS: I removed the mention of podman and docker from the README as there isn't any specific reason to use them, anything that provides a Linux shell can be used. The only reason they were previously mentioned is that they were what I used.


Add current total supply API

In order for CMC to properly display our market cap on their site, we need to provide them with a specific API endpoint that just returns the total supply as a number (and nothing else)
Tried to direct them to use our Subgraph but they don't seem to be that flexible and I can't figure out a way to get a single value out of a GraphQL URL-based query
So we can just create a simple endpoint like /supply that fetches the current total supply from the KLIMA contract ABI and returns that single value.
Original request received from CMC:

@ArchimedesCrypto I'm too lazy to sift through the contracts, what's the fastest way to determine total supply? is there a method similar to contract.getReserves() ?
@0xAurelius do I need to just return the KLIMA supply, ignoring sKLIMA?

It should just be a matter of calling the totalSupply function on the KLIMA ERC-20 contract - no need to account for sKLIMA or wsKLIMA
Example in Python for the Discord bots here: https://github.com/KlimaDAO/discord-bots/blob/main/src/contract_info.py#L36


fix: column drop position

BUG: columns are being appended to the end of a row no matter what index they are dropped at.
FIX: insert column at the correct index
current behavior

corrected behavior

:tada: This PR is included in version 1.2.3 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Updated to ASP.NET Core 2.2 Preview 3

We can experiment the new versions of dotnetcore. However, we only merge to master when the dotnet core version has been released.


Different syntax of "else"

In handlebars.js, else is implemented as "{{else}}" while in mustache-sharp is implemented as "{{#else}}", any workaround of this issue?
Moreover, in the meanwhile, I'm trying to fix this problem by using "#unless" and found that it is not supported, any possible workaround?

@joseph1125, I've implemented the unless tag as well as support for the handlebars.js if/else syntax in mosharp.
Later, Nick


Added implementation for divmod for signed ints.

Stack:

#5753
#5752
#5751
#5750
#5736 ⬅

⚠ Part of a stack created by spr. Do not merge manually using the UI - doing so may have unexpected results.

This change is

corelib/src/integer.cairo line 2154 at r2 (raw file):

                            (
                                super::downcast(q).expect('attempt to divide with overflow'),
                                super::upcast(minus(r))

Why is this negative?
Code quote:
 super::upcast(minus(r))


Control output image resolution via command line argument

Modified create_graphics.py to add an optional command line argument 'img_res' to control the resolution of output images [i.e., the dpi entry in pyplot.savefig()].

This is a great addition, Jeff! Do you have any information on how much extra time it takes to create the higher resolution image? It would be good to know if we get a request to generate higher res images in real-time.

I don't, Brian, but I could pretty easily run some timing tests to ascertain this information.

If it's not too much trouble, please add that information in this conversation before merging so we have it as a reference.  Thanks!

Thanks for the size and timing info!  That's helpful to know.
Great job!


ItemsRepeater breaks when scrolling through variable height items

Describe the bug
With a long list of elements of different heights in an ItemsRepeater, scrolling breaks, often showing a blank page.
Steps to reproduce the bug
Edit BasicDemo to contain the following:
        <controls:ItemsRepeaterScrollHost x:Name="tracker" Grid.Row="1">
            <ScrollViewer x:Name="scrollViewer">
                <controls:ItemsRepeater x:Name="repeater" VerticalCacheLength="0" Background="LightBlue">
                    <controls:ItemsRepeater.ItemTemplate>
                        <DataTemplate>
                            <TextBlock Text="{Binding Text}" Height="{Binding Height}"/>
                        </DataTemplate>
                    </controls:ItemsRepeater.ItemTemplate>
                </controls:ItemsRepeater>
            </ScrollViewer>
        </controls:ItemsRepeaterScrollHost>

    public sealed partial class BasicDemo : Page
    {
        public BasicDemo()
        {
            this.InitializeComponent();

            var random = new Random(0);
            repeater.ItemsSource = Enumerable.Range(0, 10000).Select(x => new BasicItem {
                Text = "Item " + x,
                Height = random.Next(240) + 10
            });
        }
    }

    public class BasicItem
    {
        public string Text { get; set; }
        public int Height { get; set; }
    }

Run the TestApp and use scrollbar to scroll through the list of items.
Expected behavior
Items are displayed at all positions of the scrollbar.
Screenshots

Version Info
Current master (8f497521)

Windows 10 version
Saw the problem?

Insider Build (xxxxx)

May 2019 Update (18362)
Yes

October 2018 Update (17763)

April 2018 Update (17134)

Fall Creators Update (16299)

Creators Update (15063)

Device form factor
Saw the problem?

Desktop
Yes

Mobile

Xbox

Surface Hub

IoT

This seems to happen only when using mouse thumb drag and when having a large number of elements (>~5000). Works fine for touch. It also fixes itself if you scroll by a little bit after. That suggests that this might be a bug in the disconnected jump logic in stack layout when the extent is particularly large (does not repro for say 1000 items). It is likely the size of the extent that causes it as opposed to the number of items.

I can easily reproduce this also with a much low number of items (eg. 500).
It seems to depend also on the size/height of the individual items. Or on the spread of the different item heights.
I used TextBlocks to reproduce and filled them with random sentences. Here is a screenshot of the items I used.

@agnauck please do share the repro app. Thanks.

@ranjeshj I published a demo app here:
https://github.com/agnauck/ItemsRepeaterWinUI
here you can adjust the number of items:
https://github.com/agnauck/ItemsRepeaterWinUI/blob/7b20d82c5c864ced0a601e069207d8a4cc08427d/MainWindowViewModel.cs#L27
here you can the height of the items by adding more words or sentences to the demo data generator:
https://github.com/agnauck/ItemsRepeaterWinUI/blob/master/MainWindowViewModel.cs#L34-L35

I've looked into this and here's what seems to be happening (with relevant parts if logs):

User scrolls
ViewportManagerWithPlatformFeatures::OnEffectiveViewportChanged called:

StackLayout: 	EffectiveViewportChanged event callback
StackLayout: 	Effective Viewport: (0,29960,1564,1101)->(0,355815,1564,1101).
StackLayout: 	Used Viewport: (0,29960,1564,1101)->(0,355815,1564,1101).
StackLayout: 	Invalidating measure due to viewport change.

Measure is carried out

StackLayout: 	Disconnected Window
StackLayout: 	Picked anchor:2280
StackLayout Disconnected Window - throwing away all realized elements
 GetElement: 2280 Not AutoRecycleCandidate:
StackLayout: 	Layout bounds of anchor 2280 are (0,356159,1564,26).
StackLayout: 	Generating forward from anchor 2280.

And ends expecting a viewport shift:

StackLayout: 	Extent is (1564,14199796). Based on average 142.
StackLayout Extent: (0,32399,1564,14199796).
StackLayout: 	Expecting viewport shift of (0,-31920)

Viewport shift is never recieved because we threw away all elements at the beginning of the measure, giving the Scroller nothing to anchor onto
So ViewportManagerWithPlatformFeatures::OnLayoutUpdated is called:

StackLayout: 	Layout Updated with pending shift 0 -31920- invalidating measure

And m_unshiftableShift is set to -31920
When it comes to arrange the items, ViewportManagerWithPlatformFeatures::GetLayoutVisibleWindow adds the m_unshiftableShift to the viewport
Which results in items being arranged -31920 pixels offscreen (remember our effective viewport starts at 0,355815: 355815 + -31920 == 323895):

StackLayout: 	ArrangeLayout
StackLayout: 	Arranging element 2281 at (0,323902,1564,39).
StackLayout: 	Arranging element 2282 at (0,323941,1564,239).
StackLayout: 	Arranging element 2283 at (0,324180,1564,18).
StackLayout: 	Arranging element 2284 at (0,324198,1564,199).
StackLayout: 	Arranging element 2285 at (0,324397,1564,135).
StackLayout: 	Arranging element 2286 at (0,324532,1564,101).
StackLayout: 	Arranging element 2287 at (0,324633,1564,179).
StackLayout: 	Arranging element 2288 at (0,324812,1564,110).
StackLayout: 	Arranging element 2289 at (0,324922,1564,152).
StackLayout: 	Arranging element 2290 at (0,325074,1564,162).
StackLayout: 	Arranging element 2291 at (0,325236,1564,140).

I've tried removing the code that sets m_unshiftableShift and that indeed makes scrolling work again, but only for a while: after a while of scrolling we get an unhandled exception:
Layout cycle detected.  Layout could not complete.

In addition, I think that code is needed in certain scenarios:

This can happen in cases where no scrollviewer in the parent chain can scroll in the shift direction.

I'm not sure how to proceed any further than this.

@RBrid to help investigate

I think i got same issue on my custom layout. Or very similar.

@ad1Dima If you are seeing a layout cycle crash, it is very much a layout bug. If you are seeing it in your custom layout, then the bug is likely in your custom layout. You will need to debug your layout to root cause it. It is unlikely to be fixed by a change to stack layout which this bug is talking about.

@ranjeshj this is not about layout cycle. this is about scrollview resetting it's offset to 0. This looks similar because it happens when I quickly scroll through my layout and the elements don't have time to draw.

Thanks for clarifying.

@chingucoding does some of your PRs fixing issues with items repeater / layouts fix this?

@danwalmsley I'm afraid not, the issue still persists even with the latest changes made to ItemsRepeater.
@ranjeshj I can take my shot on this if that's fine.

I think a major issue for layouting is just that the layout can't know which item should be rendered where without having rendered all predecessors as the item heights are random. So I'm not sure how much there is to gain without having to render/measure all items before hand.

Not true - AFAICS it can estimate the height of all items based on the ones currently visible and make a guess as to what to display for any particular scroll position. In fact this is working. As I mentioned in my comment if you remove the code that sets m_unshiftableShift it does work correctly for a while, until a layout cycle happens.
My problem was understanding what the "unshiftable shift" means; I assume it's there for a reason but the code just says:

This can happen in cases where no scrollviewer in the parent chain can scroll in the shift direction.

And I have no idea what that means or what could cause that. I suspect I could fix scrolling through variable height items by just removing the unshiftable shift and fixing the layout cycle problem, but I suspect that would break something else.
Is the maintainer of ItemsRepeater available to come to this thread and explain what the "unshiftable shift" does @ranjeshj ?

Not true - AFAICS it can estimate the height of all items based on the ones currently visible and make a guess as to what to display for any particular scroll position. In fact this is working. As I mentioned in my comment if you remove the code that sets m_unshiftableShift it does work correctly for a while, until a layout cycle happens.

Well it can estimate, yes agreed. But it can never know for sure, and what I noticed with the thumb drag with variable sized items is that the thumb kept jumping up and down and ending up not where the mouse cursor is, which I think is less then ideal.

Yeah that's unavoidable I think. Scroll anchoring fixes it for scrolling small increments, but for bit increments #435 would be required to prevent the thumb jumping.
In the meantime though, not displaying a blank list would be an improvement.

But it can never know for sure, and what I noticed with the thumb drag with variable sized items is that the thumb kept jumping up and down and ending up not where the mouse cursor is, which I think is less then ideal.
ListView also works like that.

I just wanted to kindly ask if there is an update or workaround for this issue. Its currently blocking my development because this issue happens in my application with more complex items a lot. But for performance reasons I need the ItemsRepeater

I'm running into this issue as well. The child items in my case are other lists, so they are varying in length and it happens so the last list has really few elements, while the ones before it have more. If I try to scroll down to the bottom the scroll position is constantly being put back up.

Does anyone have a workaround for this issue?  I have the same problem and found this issue by searching for ViewportManagerWithPlatformFeatures::OnLayoutUpdated which is where I discovered the m_unshiftableShift causing our issues.
I have a demo app that exhibits the same problem https://github.com/kokuda/ItemsRepeaterShiftedLayoutExample.
The problem fixes itself with some user input, is there some user input we can fake that will break the ItemsRepeater out of the m_unshiftableShift state?

I haven't tested it, but I expect the layout cycle crash should be fixed in 1.5 via the fix for #6218.


Update and complete L11 metadata

The latest patent on L11 metadata defines the block more precisely:
// byte 0
content_type           u(4)
content_sub_type       u(4)

// byte 1
white_point            u(4)
reference_mode_flag    u(1)
reserved               u(3)

// byte 2
sharpness              u(2)
noise_reduction        u(2)
mpeg_noise_reduction   u(2)
frame_rate_conversion  u(2)

// byte 3
brightness             u(2)
color                  u(2)
reserved1              u(2)
reserved2              u(2)

However I'm not sure that the existing implementations actually behave correctly according to these changes.
For example, increasing the intended whitepoint never becomes warmer, always colder.
While it's specified that 0 is default (no changes) and 4 is now D65.

Current byte 0 and byte 1 implementations should be correct. The byte 2 is called Content adjustment in latest verifier. Some Android devices supporting Dolby Vision recording set this byte to 8, however we don't know the meaning of it in actual implementations. The byte 3 should be still reserved.

Right.
Both content type and white point don't match the new patent in the verifier.
The Content adjustment summary isn't really useful.


Quantized vs float comparison in TfLite MobileNet V1

Dear all,
I performed some benchmarks comparing quantized models versus float models for different versions of  MobileNet V1 TfLite model. The hardware I used is a Freescale IMX6Q board with 4 cortex A9 @ 1 GHz. I used the master branch of armnn and ComputeLibrary downloaded from review.mlplatform.org and compiled with only NEON aceleration. I downloaded all the models from the official TensorflowLite repository.
The table below shows the results, where the second column is the processing time for quantized models in ms, the third column is the processing time for float models in ms and the last column is the time boost factor of the quantized model compared to the float one.
The expectation was that the quantized models run faster than the float models, but instead is the opposite. What is the reason of these results? Are the results reasonable?
Thanks
MobileNet_v1_1.0_224 | 990 | 406 | 2,43842365
MobileNet_v1_1.0_192 | 720 | 263 | 2,73764259
MobileNet_v1_1.0_160 | 527 | 204 | 2,58333333
MobileNet_v1_1.0_128 | 262 | 142 | 1,84507042
MobileNet_v1_0.75_224 | 768 | 262 | 2,93129771
MobileNet_v1_0.75_192 | 554 | 173 | 3,20231214
MobileNet_v1_0.75_160 | 405 | 131 | 3,09160305
MobileNet_v1_0.75_128 | 216 | 91 | 2,37362637
MobileNet_v1_0.50_224 | 442 | 144 | 3,06944444
MobileNet_v1_0.50_192 | 324 | 98 | 3,30612245
MobileNet_v1_0.50_160 | 232 | 74 | 3,13513514
MobileNet_v1_0.50_128 | 119 | 57 | 2,0877193
MobileNet_v1_0.25_224 | 219 | 64 | 3,421875
MobileNet_v1_0.25_192 | 162 | 46 | 3,52173913
MobileNet_v1_0.25_160 | 117 | 36 | 3,25
MobileNet_v1_0.25_128 | 63 | 26 | 2,42307692

Hello @liviolima80,
Suppose this is for armv7 architecture. That should explain the performance difference as we don't have optimized assembly kernels for quantized kernels for armv7 while we do for fp32. We are planning to include some in the near future.
Hope this helps.

Likewise, I'm unable to obtain any speedup on quantized TFLite MobileNet v1 1.0 on aarch64, running on Odroid C2. Quantized model is slower, though by nowhere near as much. For comparison, TFLite itself cranks out 15.44 inferences per second on this board, compared to ARM NN's 9.96 inferences per second in quantized mode. I'm kind of wondering if I'm "holding it wrong".

@depthwise every board is different, but I'm getting figures around 70-80ms per inference for Mobilenet v1 1.0 224, with 8-bit quantized format, for 4x A53 @1.8Ghz in the latest master of ArmNN.
The quantized model is slightly faster than the floating point model for me, but not by much. The Cortex-A53 has pretty good floating-point support, and lacks the 8-bit dot-product instruction that you get on recent big CPUs, also Mobilenet is not memory-bound anywhere. Slightly faster but not massively so would be expected in this case.
Hope this helps,
Matthew

Thanks for the confirmation. There's always a risk that I'm "holding it wrong", but it seems like your own testing confirms my suspicions.
The good news (for ARM) is, there seems to be some headroom remaining then. TFLite is not known for highly optimized implementations of anything, and they manage to be substantially faster on the same exact model on the same exact board, even though they're way slower on fp32. Knowing what you know about ARM architecture, you folks gotta be able to beat them. :1st_place_medal:
P.S. I'm also using the tip-of-tree version of ARM NN from cgit repo.

OK thanks @depthwise I'll pass your feedback on internally. If Tensorflow Lite can hit 15.44 inferences / second (about 65ms?) then there must be something that we can do to at least match it.

I've just checked and there will be work on int8 performance for CPU in the near future.

I've just checked and there will be work on int8 performance for CPU in the near future.

My experiment on a moderate ARM64 device (Xiaomi A2 phone) shows around 200ms for float model mobilenet_v1_1.0_224.tflite, and 130ms for same, converted with --post_training_quantize. Enabling of NNAPI reverses the order: 120ms for float, 130ms for quantized.
What is more puzzling, same model with quantized input works slower: 185ms (it fails with NNAPI, but that's a different story). This is the convert command:
tflite_convert --output_file=mobilenet_v1_1.0_224_qinput.tflite --graph_def_file=mobilenet_v1_1.0_224_frozen.pb --inference_input_type=QUANTIZED_UINT8 --input_arrays=input --output_arrays=MobilenetV1/Predictions/Reshape_1 --std_dev_values=255 --mean_values=0 --default_ranges_min=0 --default_ranges_max=7


Electron tests: model export

Exporting a model should save it to the disk.

Create a new project
Click on the project
Add KCL
Click on "Export current part"
Select gltf as the type
Save
Check the file size matches exactly what we expect.

We have export tests for the web app where after we do the export we send the file to the CLI for a screen shot, I think we should do the same here instead of matching the file size, as it's both a better saniy check and should be less brittle.
If David changes something in the header than the file size is going to change.


Question on how to configure channel slider

This question builds off a previous issue #1535.
I have some multi-channel fluorescence data and several bit-mask annotations. I put all the masks together in one ome-tiff, which works well to control the color and visibility of the annotations. You can see my toy config here. The small issue I have is that the slider for each annotation channel starts at [1,1], so nothing is visible until I move it to [0,1]. I'm worried other users wouldn't think to do this.
I've tried setting the slider using coordinationSpace, but then none of the annotations appear. I've also tried configuring the annotation ome-tiff with "isBitmask": true, but the problem there is that I don't know how to control/change the color of each channel (they're all the same color). What's the best solution here?

I have this figured out now. Some takeaways: our typical dataset contains ~100k cells, and ome-tiff bitmasks tended to crash the app (deployed through React). I have 6 channels of annotations and there was a decent amount of empty space in the images (not sure if that matters). Using zarr made everything work much faster and I don't have to do any cropping.


fix: Error message conversion specifier

%d was used on the gateway error message log instead of %s also added clarity as to the application and origin of the error message

Please follow the convention used throughout notecard and Notehub, which is to keep it lowercase and have simple messages concatenation by ": "
meaning, even though %d needed to be changed to %s, there was no reason to change ": " to "ERROR --> " which is an inconsistent style.

Now following string error response style from:
https://github.com/blues/sparrow-lora/blob/master/Application/Framework/dfu.c#L143