Use SQLite for parts database

The intent here is to use this with https://github.com/phiresky/sql.js-httpvfs. There are major missing pieces:

the python sqlite on my system doesn't have the fts5 extension availible, so while sql.js-httpvfs theoretically supports full-text search, it is not available
the sql.js-httpvfs VFS needs to be modified to download the index file and map between the index & the gzipped data, and use the Compression Streams API to decompress each chunk
no work was done on the frontend portion of jlcparts

Quick benchmarks:

algo
page size (KiB)
zstd dict (KiB)
compressed (MiB)
uncompressed (MiB)
ratio
cpu (s)
compression cpu (s)

zstd
1
8
111
537
21
99
45

zstd
2
8
96
566
17
82
28

zstd
4
8
74
634
12
74
20

zstd
8
8
54
545
10
64
10

zstd
16
8
42
513
8
62
8

zstd
1
16
104
537
19
130
76

zstd
2
16
89
566
16
98
44

zstd
4
16
71
634
11
81
27

zstd
8
16
51
545
9
68
14

zstd
16
16
41
513
8
62
8

zstd
1
32
98
537
18
205
151

zstd
2
32
80
566
14
139
85

zstd
4
32
68
634
11
103
49

zstd
8
32
49
545
9
78
24

zstd
16
32
40
513
8
67
13

zstd
1
64
87
537
16
313
259

zstd
2
64
73
566
13
202
148

zstd
4
64
59
634
9
136
82

zstd
8
64
46
545
8
93
39

zstd
16
64
38
513
7
75
21

zstd
1
128
85
537
16
2696
2642

zstd
2
128
71
566
13
1452
1398

zstd
4
128
58
634
9
765
711

zstd
8
128
46
545
8
392
338

zstd
16
128
38
513
7
206
152

zstd
1
256
84
537
16
4318
4264

zstd
2
256
70
566
12
2101
2047

zstd
4
256
58
634
9
1115
1061

zstd
8
256
51
545
9
841
787

zstd
16
256
55
513
11
60
6

none
1

537

54
0

none
2

566

53
0

none
4

634

54
0

none
8

545

53
0

none
16

513

54
0

gzip
1

198
537
37
83
29

gzip
2

164
566
29
69
15

gzip
4

122
634
19
66
12

gzip
8

78
545
14
61
7

gzip
16

56
513
11
62
8

note: not super helpful in the CPU & time area because I have not tested read performance.
See https://github.com/yaqwsx/jlcparts/issues/37

I've updated the above with some more data.

generated with this test script
#!/bin/zsh

for dict_size in 8192 16384 32768 65536<PHONE_NUMBER>44
do
for page_size in 1024 2048 4096 8192 16384
do
    time jlcparts buildtables --jobs 16 --ignoreoldstock 30 --pagesize "$page_size" --compression-algorithm zstd --dictsize $dict_size cache.sqlite3 web/public/data > /dev/null
    sqlite_size=$(ls -lh web/public/data/index.sqlite3 | awk '{print $5}')
    index_size=$(ls -lh web/public/data/index.sqlite3.zstindex | awk '{print $5}')
    parts_size=$(ls -lh web/public/data/index.sqlite3.zstpart | awk '{print $5}')
    echo "zstd, dict_size=${dict_size} page_size=${page_size}: (sqlite3=${sqlite_size}, index=${index_size}, parts=${parts_size})"
done
done

for page_size in 1024 2048 4096 8192 16384
do
    time jlcparts buildtables --jobs 16 --ignoreoldstock 30 --pagesize "$page_size" --compression-algorithm gzip cache.sqlite3 web/public/data > /dev/null
    sqlite_size=$(ls -lh web/public/data/index.sqlite3 | awk '{print $5}')
    index_size=$(ls -lh web/public/data/index.sqlite3.gzindex | awk '{print $5}')
    parts_size=$(ls -lh web/public/data/index.sqlite3.gzpart | awk '{print $5}')
    echo "gzip, page_size=${page_size}: (sqlite3=${sqlite_size}, index=${index_size}, parts=${parts_size})"
done

for page_size in 1024 2048 4096 8192 16384
do
    time jlcparts buildtables --jobs 16 --ignoreoldstock 30 --pagesize "$page_size" --compression-algorithm none cache.sqlite3 web/public/data > /dev/null
    sqlite_size=$(ls -lh web/public/data/index.sqlite3 | awk '{print $5}')
    echo "none, page_size=${page_size}: (sqlite3=${sqlite_size})"
done

data:

algo=zstd, dict_size=8192, page_size=1024, sqlite3=537M, index=4.2M, parts=107M, user=98.74s, system=44.37s, cpu=490%, total=29.195
algo=zstd, dict_size=8192, page_size=2048, sqlite3=566M, index=2.3M, parts=94M, user=81.53s, system=38.07s, cpu=460%, total=25.964
algo=zstd, dict_size=8192, page_size=4096, sqlite3=634M, index=1.3M, parts=73M, user=73.70s, system=33.54s, cpu=424%, total=25.261
algo=zstd, dict_size=8192, page_size=8192, sqlite3=545M, index=553K, parts=53M, user=64.36s, system=29.37s, cpu=432%, total=21.670
algo=zstd, dict_size=8192, page_size=16384, sqlite3=513M, index=265K, parts=42M, user=61.74s, system=27.98s, cpu=421%, total=21.286
algo=zstd, dict_size=16384, page_size=1024, sqlite3=537M, index=4.3M, parts=100M, user=130.27s, system=46.43s, cpu=585%, total=30.160
algo=zstd, dict_size=16384, page_size=2048, sqlite3=566M, index=2.3M, parts=87M, user=97.57s, system=38.20s, cpu=519%, total=26.115
algo=zstd, dict_size=16384, page_size=4096, sqlite3=634M, index=1.3M, parts=70M, user=81.39s, system=33.74s, cpu=481%, total=23.913
algo=zstd, dict_size=16384, page_size=8192, sqlite3=545M, index=561K, parts=50M, user=67.96s, system=29.74s, cpu=448%, total=21.759
algo=zstd, dict_size=16384, page_size=16384, sqlite3=513M, index=273K, parts=41M, user=62.47s, system=27.97s, cpu=434%, total=20.835
algo=zstd, dict_size=32768, page_size=1024, sqlite3=537M, index=4.3M, parts=94M, user=204.76s, system=51.62s, cpu=727%, total=35.258
algo=zstd, dict_size=32768, page_size=2048, sqlite3=566M, index=2.3M, parts=78M, user=138.59s, system=40.91s, cpu=618%, total=29.011
algo=zstd, dict_size=32768, page_size=4096, sqlite3=634M, index=1.3M, parts=67M, user=103.12s, system=35.89s, cpu=541%, total=25.678
algo=zstd, dict_size=32768, page_size=8192, sqlite3=545M, index=577K, parts=48M, user=78.08s, system=30.13s, cpu=486%, total=22.225
algo=zstd, dict_size=32768, page_size=16384, sqlite3=513M, index=289K, parts=40M, user=67.19s, system=28.27s, cpu=448%, total=21.289
algo=zstd, dict_size=65536, page_size=1024, sqlite3=537M, index=4.3M, parts=83M, user=312.52s, system=55.79s, cpu=865%, total=42.561
algo=zstd, dict_size=65536, page_size=2048, sqlite3=566M, index=2.3M, parts=71M, user=201.91s, system=43.34s, cpu=733%, total=33.439
algo=zstd, dict_size=65536, page_size=4096, sqlite3=634M, index=1.4M, parts=58M, user=135.94s, system=36.79s, cpu=624%, total=27.670
algo=zstd, dict_size=65536, page_size=8192, sqlite3=545M, index=609K, parts=45M, user=92.84s, system=31.08s, cpu=537%, total=23.035
algo=zstd, dict_size=65536, page_size=16384, sqlite3=513M, index=321K, parts=38M, user=74.90s, system=28.70s, cpu=480%, total=21.552
algo=zstd, dict_size=131072, page_size=1024, sqlite3=537M, index=4.4M, parts=81M, user=2696.38s, system=118.30s, cpu=1388%, total=3:22.65
algo=zstd, dict_size=131072, page_size=2048, sqlite3=566M, index=2.4M, parts=69M, user=1451.82s, system=77.68s, cpu=1297%, total=1:57.84
algo=zstd, dict_size=131072, page_size=4096, sqlite3=634M, index=1.4M, parts=57M, user=765.11s, system=55.65s, cpu=1173%, total=1:09.96
algo=zstd, dict_size=131072, page_size=8192, sqlite3=545M, index=673K, parts=45M, user=392.49s, system=38.96s, cpu=997%, total=43.261
algo=zstd, dict_size=131072, page_size=16384, sqlite3=513M, index=385K, parts=38M, user=205.56s, system=34.54s, cpu=795%, total=30.165
algo=zstd, dict_size=262144, page_size=1024, sqlite3=537M, index=4.5M, parts=79M, user=4317.83s, system=129.94s, cpu=1389%, total=5:20.08
algo=zstd, dict_size=262144, page_size=2048, sqlite3=566M, index=2.5M, parts=67M, user=2100.99s, system=80.78s, cpu=1330%, total=2:43.97
algo=zstd, dict_size=262144, page_size=4096, sqlite3=634M, index=1.5M, parts=56M, user=1114.82s, system=58.12s, cpu=1193%, total=1:38.27
algo=zstd, dict_size=262144, page_size=8192, sqlite3=545M, index=791K, parts=50M, user=840.95s, system=44.49s, cpu=1171%, total=1:15.61
algo=zstd, dict_size=262144, page_size=16384, sqlite3=513M, index=257K, parts=55M, user=59.91s, system=28.42s, cpu=416%, total=21.187
algo=none, page_size=1024, sqlite3=537M, user=54.11s, system=28.21s, cpu=357%, total=23.002
algo=none, page_size=2048, sqlite3=566M, user=53.32s, system=27.39s, cpu=371%, total=21.723
algo=none, page_size=4096, sqlite3=634M, user=53.80s, system=27.15s, cpu=377%, total=21.421
algo=none, page_size=8192, sqlite3=545M, user=53.45s, system=25.91s, cpu=397%, total=19.955
algo=none, page_size=16384, sqlite3=513M, user=53.69s, system=25.61s, cpu=406%, total=19.505
algo=gzip, page_size=1024, sqlite3=537M, index=4.2M, parts=194M, user=83.26s, system=38.45s, cpu=242%, total=50.192
algo=gzip, page_size=2048, sqlite3=566M, index=2.3M, parts=162M, user=69.16s, system=30.83s, cpu=279%, total=35.818
algo=gzip, page_size=4096, sqlite3=634M, index=1.3M, parts=121M, user=65.55s, system=29.80s, cpu=298%, total=31.991
algo=gzip, page_size=8192, sqlite3=545M, index=545K, parts=77M, user=60.83s, system=27.76s, cpu=324%, total=27.329
algo=gzip, page_size=16384, sqlite3=513M, index=257K, parts=56M, user=62.02s, system=27.45s, cpu=321%, total=27.828

algo
cpu
dict_size
index (MiB)
page_size
parts (MiB)
sqlite3 (MiB)
system (s)
total (s)
user (s)
page size (KiB)
zstd dict (KiB)

zstd
490%
8192
4.2
1024
107
537
44
29
99
1
8

zstd
460%
8192
2.3
2048
94
566
38
26
82
2
8

zstd
424%
8192
1.3
4096
73
634
34
25
74
4
8

zstd
432%
8192
0.5
8192
53
545
29
22
64
8
8

zstd
421%
8192
0.3
16384
42
513
28
21
62
16
8

zstd
585%
16384
4.3
1024
100
537
46
30
130
1
16

zstd
519%
16384
2.3
2048
87
566
38
26
98
2
16

zstd
481%
16384
1.3
4096
70
634
34
24
81
4
16

zstd
448%
16384
0.5
8192
50
545
30
22
68
8
16

zstd
434%
16384
0.3
16384
41
513
28
21
62
16
16

zstd
727%
32768
4.3
1024
94
537
52
35
205
1
32

zstd
618%
32768
2.3
2048
78
566
41
29
139
2
32

zstd
541%
32768
1.3
4096
67
634
36
26
103
4
32

zstd
486%
32768
0.6
8192
48
545
30
22
78
8
32

zstd
448%
32768
0.3
16384
40
513
28
21
67
16
32

zstd
865%
65536
4.3
1024
83
537
56
43
313
1
64

zstd
733%
65536
2.3
2048
71
566
43
33
202
2
64

zstd
624%
65536
1.4
4096
58
634
37
28
136
4
64

zstd
537%
65536
0.6
8192
45
545
31
23
93
8
64

zstd
480%
65536
0.3
16384
38
513
29
22
75
16
64

zstd
1388%
131072
4.4
1024
81
537
118
203
2696
1
128

zstd
1297%
131072
2.4
2048
69
566
78
118
1452
2
128

zstd
1173%
131072
1.4
4096
57
634
56
70
765
4
128

zstd
997%
131072
0.7
8192
45
545
39
43
392
8
128

zstd
795%
131072
0.4
16384
38
513
35
30
206
16
128

zstd
1389%
262144
4.5
1024
79
537
130
320
4318
1
256

zstd
1330%
262144
2.5
2048
67
566
81
164
2101
2
256

zstd
1193%
262144
1.5
4096
56
634
58
98
1115
4
256

zstd
1171%
262144
0.8
8192
50
545
44
76
841
8
256

zstd
416%
262144
0.3
16384
55
513
28
21
60
16
256

none
357%

1024

537
28
23
54
1

none
371%

2048

566
27
22
53
2

none
377%

4096

634
27
21
54
4

none
397%

8192

545
26
20
53
8

none
406%

16384

513
26
20
54
16

gzip
242%

4.2
1024
194
537
38
50
83
1

gzip
279%

2.3
2048
162
566
31
36
69
2

gzip
298%

1.3
4096
121
634
30
32
66
4

gzip
324%

0.5
8192
77
545
28
27
61
8

gzip
321%

0.3
16384
56
513
27
28
62
16

NOTE: this time is compression time, not decompression time

seems to me like zstd with 64KiB dict & 8K pages is likely the best tradeoff here. zstd has good decompression speed across a range of compression levels: https://github.com/facebook/zstd#benchmarks

Thank you for your input. I really appreciate it. I thought about using SQLite for a while, but I never found time (and motivation) to do it. Let me share my findings:

Originally, I wanted to serve the SQLite over HTTPVFS. After doing several tests I abandoned the idea - most users use the full-text feature. It is very easy to run into one of the worst-case complexity scenarios that leads to fetching a non-trivial part of the database. This leads to fetching quite a lot of data when using the site. Thus, I think a better solution is to store a compressed database in local storage and use SQLite over a compressed filesystem. The usage of the site will remain roughly the same - the user will be notified about new database which they can fetch. The fetching will be fairly quick as we need to just download the image and do nothing else (96% of the current update time is the update of entries in IndexedDB).
next, I was thinking about a suitable DB schema. Careful design can save a lot of DB size. Meanwhile representation of the categories is trivial, I struggled with a suitable design for attributes as they are non-uniform. I haven't found a design that I was satisfied with. I would have to dig up my old notes to give you more details.
I want to connect this huge change with a rewrite of the frontend into React functional components and refactor it. This project was my first project in React, and the frontend is quite messy. I would also like to encode the current search query into the URL so the users can use bookmarks to create a shortcuts.

The usage of the site will remain roughly the same - the user will be notified about new database which they can fetch

I find this part of the site particularly frustrating. I don't like having to constantly refresh all the data and have to think about that, I'd rather that just get handled seamlessly in the background.

most users use the full-text feature. It is very easy to run into one of the worst-case complexity scenarios that leads to fetching a non-trivial part of the database.

Interesting. The full-text feature definitely needs to be investigated further. I don't know about SQLite, but with Postgresql, there's things like covering indexes and index-only scans. I wonder if it'd be possible to get that in SQLite FTS.
I'm not familiar with the existing code, but there's also plenty of knobs around tokenizing the query, stop words, throttling delay, limiting result counts, etc.
If you have your old notes, I'd be interested in reading them.

Thus, I think a better solution is to store a compressed database in local storage and use SQLite over a compressed filesystem. The fetching will be fairly quick as we need to just download the image and do nothing else (96% of the current update time is the update of entries in IndexedDB).

Not opposed to this, but I don't see why it couldn't be both: fetch chunks only as they are needed, and cache them locally. In fact, I wonder if web browsers already handle caching internally these days.

next, I was thinking about a suitable DB schema. Careful design can save a lot of DB size. Meanwhile representation of the categories is trivial, I struggled with a suitable design for attributes as they are non-uniform.

I stuck with the existing IndexedDB schema design in my testing. But yes, there is plenty to look at in terms of structuring the data so it can be queried easily.

I want to connect this huge change with a rewrite of the frontend into React functional components and refactor it.

I've often felt this same temptation to combine two major changes into one :smiley:. I've found it ends up more motivating and more efficient for me to do changes in smaller chunks, even if at times it seems like I'm doing work that I will soon replace.

There's many config options for full-text-search with sqlite. I don't remember if I wrote my research about this down somewhere but you can reduce the FTS size by >90% by setting detail=none and contentless (https://www.sqlite.org/fts5.html#the_detail_option). It does reduce the power of the queries you can do a lot though.
Also note that sqlite is very easy to compile and there's also drop in python packages to get a newer and more complete sqlite into python.
There's one powerful statically hosted full text search engine that scales to a terabyte of data that i know of called summa, but for an index with a compressed size of 50MB it's probably not worth it.
There's a few other JS libraries that allow creating a keyword / full text search index that you serialize to JSON that you fully download (with a smaller size) and you could then use in combination with SQLite or something else to fetch the full data dynamically. One example (I think) is https://github.com/nextapps-de/flexsearch

Also just as a note if you have too much free time: If you download the whole DB then you can alternatively also create a minimal db without indexes and without FTS, download that, and create the indexes and FTS search locally. Trading bandwidth for local compute.


[enhancement request] Floating window + shortcut

It would be nice if there was an option to enable fromscratch to be a floating window and also trigger it by using a shortcut, I think this really makes sense in a scratchpad app, since normally you will need to open it quickly in your current workspace to write something and then hide it and continue doing you previous task.
This functionality I'm describing is the same as in Numi: https://numi.io

I like that. What shortcut does Numi use for quickview, and what exactly is a "floating window"? (on Ubuntu, which I use, I can set arbitrary windows to "always stay on top")

I installed Numi, and it didn't set a default shortcut for bringing it to the foreground.

I was just going to make the same suggestion that @jcalonso submitted. A floating window is one that will stay in front of all other normal windows even when the floating window doesn't have focus.
The FromScratch usage in this case would be that your notes would always be accessible as you switch between other apps. When you wanted it out of the way you would hit the hot key to hide it. This is particularly useful when working on a laptop with no additional screens.
As it is now to get access to your FromScratch text in multiple apps, you've got to switch to the app and then switch to FromScratch to bring it to the front. Some apps provide a preference to make things like tool palettes floating windows so they never get obscured.

I understand the use case and I use it a lot myself, but floating windows is something that should be done on OS level, not on app level. There's a bunch of window management options for all three OSses FromScratch runs on that offer this.

CMD/CTRL+F?
you could make it so that it fades away after a certain amount of inactivity... like electron 'ClickThrough' property.

We already have that reserved for search/find :) The fading away is also something your window manager could/should do (compiz does this beautifully, for example)

This thread was updated long ago. Just wanted to know if some work is being done in this direction.
I also expected a hot-key toggle for scratch-pad. I use Guake on ubuntu 18.04 as my primary terminal application. It is a drop-down terminal which toggles visibililty on press of hot-key as "F12". It has this feature similar to iTerm2 on MacOs. I am addicted to this kind of toggle and trying hard to have this functionality work on my system using some ubuntu window management tools like wmctrl and xdotool. I am just curious to know if there is a possibility to add this functionality out-of-the-box.


Update __kube_prompt.fish added absolute pat for test and stat commands

must be tested with linux, but I think it's ok. https://github.com/aluxian/fish-kube-prompt/issues/1

@ulidtko the initial issue (by 3 years by now) was this: https://github.com/aluxian/fish-kube-prompt/issues/1 and it was related to different versions of the test command present in my environment.

Thanks for the reply. Since #11 has landed, I believe the initial issue should be resolved.


[LITE] INT8 and INT16 types support for Tile op

Closing PR this as the changes have been moved here #28416


Zoom and Drag with d3.geo.projections v4

I can't seem to get d3 zoom to work properly with d3 geo projections.
I've converted  Ivy Wang's Drag to Rotate the Globe to v4 and combined it with zoom. It works properly with mouse functions as long as I include  svg.on("mousedown.zoom", null); , otherwise d3.zoom mousedown overrides the d3.drag functions. I still can't get it to work with touch. Double tapping and two finger zooming works on a laptop (Asus, windows 10, google chrome), but on mobile (IOS 11.4) it only changes lineweights I've included in my zoom function but doesn't scale. Also, rotating the projection doesn't work unless I include
   svg.on("mousedown.zoom", null)
      .on("touchstart.zoom", null)
      .on("touchmove.zoom", null)
      .on("touchend.zoom", null);

which disables touch zooming all together. These things seem to work together on Jason Davies site though when you try to zoom on mobile it usually zooms the entire page. I would love to know if I am relatively close to getting this to work, or if it simply won't work combining these zoom and drag functions. Is there a working example of this somewhere? I haven't been able to find anything in v4 with map projections and that works on touch.
My whole project is on my Codepen. Zoom function start on line 287 and drag on line 344. Here's a code snippet:
var svg = d3
  .select("body")
  .append("svg")
  .attr("id", "world")
  .attr("width", width)
  .attr("height", height)
  .call(d3.zoom().on("zoom", zoomed));

 svg.on("mousedown.zoom", null)
    .on("touchstart.zoom", null)
    .on("touchmove.zoom", null)
    .on("touchend.zoom", null);

function zoomed() {
   let k = d3.event.transform.k;
   d3.selectAll("#world").attr("transform", "scale(" + k + ")");
   d3.selectAll(".graticule").attr("stroke-width", 0.5 / k);
   d3.selectAll(".pin").attr("stroke-width", 2 / k);
   d3.selectAll(".travelPath").attr("stroke-width", 2 / k);
   d3.selectAll(".land").attr("stroke-width", 1 / k);
   d3.selectAll(".border").attr("stroke-width", 1 / k);
 }

There’s an up-to-date Versor Dragging example here:
https://observablehq.com/@d3/versor-dragging
It’d be nice to improve it by adding zooming. I’m not sure this would qualify for using d3.zoom, however, as the logic for panning and zooming a globe is very different from panning and zooming on the plane. Maybe take a look at d3-geo-zoom?


ExprMatch does not have a Parse impl

In order to parse a match expression, you have to parse an expression and then check if it's a match expression. If you are expecting a match expression then just being able to parse that would give better errors and be simpler.

Thanks, I would accept a PR to add this impl.


Add optional ramping constraints

See #581 for details.

[x] Add variable
[x] Add parameters
[x] Add methods (sets)

Filtered all assets - did not specify it must be producer/conversion

[x] Add expression
[ ] Add constraints
[ ] Change constraint
[ ] Performance test the activation method
[ ] Check & avoid redundant constraints

[ ] If ramp limit > (max capacity - min capacity), no constraint
[ ] Others?

[x] Add params to Norse, Storage, Tiny, Variable Resolution, benchmark EU

[x] unit_commitment
[x] unit_commitment_method
[x] min_oper_point
[x] max_ramp_up
[x] max_ramp_down

[x] Add params to import schema

[x] unit_commitment
[x] unit_commitment_method
[x] min_oper_point
[x] max_ramp_up
[x] max_ramp_down

[x] Add params to structures.jl (as necessary?)
~- [ ] unit_commitment~
~- [ ] unit_commitment_method~

[x] min_oper_point
[x] max_ramp_up
[x] max_ramp_down

Not sure all the places the params need to be listed. I would assume unit_commitment and unit_commitment_method wouldn't really need to go into the graph, since they're constraint switches. But anyway the schema/structure/graph/df thing is annoying.


[Snyk] Fix for 3 vulnerabilities

This PR was automatically created by Snyk using the credentials of a real user.![snyk-top-banner](https://github.com/andygongea/OWASP-Benchmark/assets/818805/c518c423-16fe-447e-b67f-ad5a49b5d123)
Snyk has created this PR to fix 3 vulnerabilities in the npm dependencies of this project.
Snyk changed the following file(s):

packages/node/base/package.json
packages/node/base/package-lock.json

Vulnerabilities that will be fixed with an upgrade:

Issue
Score

Regular Expression Denial of Service (ReDoS) SNYK-JS-SEMVER-3247795
  169

Missing Release of Resource after Effective Lifetime SNYK-JS-INFLIGHT-6095116
  141

Regular Expression Denial of Service (ReDoS) SNYK-JS-MINIMATCH-3050818
  45

[!IMPORTANT]

Check the changes in this PR to ensure they won't cause issues with your project.
Max score is 1000. Note that the real score may have changed since the PR was raised.
This PR was automatically created by Snyk using the credentials of a real user.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information:
🧐 View latest project report
📜 Customise PR templates
🛠 Adjust project settings
📚 Read about Snyk's upgrade logic

Learn how to fix vulnerabilities with free interactive lessons:
🦉 Regular Expression Denial of Service (ReDoS)

Checkmarx One – Scan Summary & Details – a26f5eb2-8b20-4779-9e2d-d00d9e72dbaa
New Issues

Severity
Issue
Source File / Package
Checkmarx Insight

Aurora With Disabled at Rest Encryption
/rds.tf: [1](https://github.com/ACN-APPSAS/terragoat/blob/snyk-fix-f72334b67a62135581b29795d4e5b0e5//terraform/aws/rds.tf# L1)
Amazon Aurora does not have encryption for data at rest enabled. To prevent such a scenario, update the attribute 'StorageEncrypted' to 'true'.

Aurora With Disabled at Rest Encryption
/rds.tf: [97](https://github.com/ACN-APPSAS/terragoat/blob/snyk-fix-f72334b67a62135581b29795d4e5b0e5//terraform/aws/rds.tf# L97)
Amazon Aurora does not have encryption for data at rest enabled. To prevent such a scenario, update the attribute 'StorageEncrypted' to 'true'.

Aurora With Disabled at Rest Encryption
/rds.tf: [65](https://github.com/ACN-APPSAS/terragoat/blob/snyk-fix-f72334b67a62135581b29795d4e5b0e5//terraform/aws/rds.tf# L65)
Amazon Aurora does not have encryption for data at rest enabled. To prevent such a scenario, update the attribute 'StorageEncrypted' to 'true'.

Aurora With Disabled at Rest Encryption
/rds.tf: [130](https://github.com/ACN-APPSAS/terragoat/blob/snyk-fix-f72334b67a62135581b29795d4e5b0e5//terraform/aws/rds.tf# L130)
Amazon Aurora does not have encryption for data at rest enabled. To prevent such a scenario, update the attribute 'StorageEncrypted' to 'true'.

Aurora With Disabled at Rest Encryption
/rds.tf: [113](https://github.com/ACN-APPSAS/terragoat/blob/snyk-fix-f72334b67a62135581b29795d4e5b0e5//terraform/aws/rds.tf# L113)
Amazon Aurora does not have encryption for data at rest enabled. To prevent such a scenario, update the attribute 'StorageEncrypted' to 'true'.

Aurora With Disabled at Rest Encryption
/rds.tf: [81](https://github.com/ACN-APPSAS/terragoat/blob/snyk-fix-f72334b67a62135581b29795d4e5b0e5//terraform/aws/rds.tf# L81)
Amazon Aurora does not have encryption for data at rest enabled. To prevent such a scenario, update the attribute 'StorageEncrypted' to 'true'.

Aurora With Disabled at Rest Encryption
/rds.tf: [33](https://github.com/ACN-APPSAS/terragoat/blob/snyk-fix-f72334b67a62135581b29795d4e5b0e5//terraform/aws/rds.tf# L33)
Amazon Aurora does not have encryption for data at rest enabled. To prevent such a scenario, update the attribute 'StorageEncrypted' to 'true'.

Aurora With Disabled at Rest Encryption
/rds.tf: [17](https://github.com/ACN-APPSAS/terragoat/blob/snyk-fix-f72334b67a62135581b29795d4e5b0e5//terraform/aws/rds.tf# L17)
Amazon Aurora does not have encryption for data at rest enabled. To prevent such a scenario, update the attribute 'StorageEncrypted' to 'true'.

Aurora With Disabled at Rest Encryption
/rds.tf: [49](https://github.com/ACN-APPSAS/terragoat/blob/snyk-fix-f72334b67a62135581b29795d4e5b0e5//terraform/aws/rds.tf# L49)
Amazon Aurora does not have encryption for data at rest enabled. To prevent such a scenario, update the attribute 'StorageEncrypted' to 'true'.

CVE-2023-42282
Npm-ip-1.1.5
Vulnerable Package

CVE-2023-45133
<EMAIL_ADDRESS>Vulnerable Package

CVE-2023-45133
<EMAIL_ADDRESS>Vulnerable Package

CVE-2024-27088
Npm-es5-ext-0.10.53
Vulnerable Package

CVE-2024-27351
Python-Django-1.2
Vulnerable Package

CVE-2024-29180
Npm-webpack-dev-middleware-5.2.1
Vulnerable Package

CVE-2024-4068
Npm-braces-2.3.2
Vulnerable Package

CVE-2024-4068
Npm-braces-3.0.2
Vulnerable Package

Cx37f28a21-b2a7
Npm-angular-1.8-1.8.2
Vulnerable Package

Cx94a5c81d-63b1
Npm-angular-1.8-1.8.2
Vulnerable Package

Cx9eb96c9b-eb45
Npm-angular-mocks-1.8-1.8.2
Vulnerable Package

Cxcf770dcb-fd21
Npm-angular-1.8-1.8.2
Vulnerable Package

Passwords And Secrets - Generic Access Key
/lambda.tf: [45](https://github.com/ACN-APPSAS/terragoat/blob/snyk-fix-f72334b67a62135581b29795d4e5b0e5//terraform/aws/lambda.tf# L45)
Query to find passwords and secrets in infrastructure code.

Passwords And Secrets - Generic Access Key
/providers.tf: [10](https://github.com/ACN-APPSAS/terragoat/blob/snyk-fix-f72334b67a62135581b29795d4e5b0e5//terraform/aws/providers.tf# L10)
Query to find passwords and secrets in infrastructure code.

Passwords And Secrets - Generic Password
/sql.tf: [50](https://github.com/ACN-APPSAS/terragoat/blob/snyk-fix-f72334b67a62135581b29795d4e5b0e5//terraform/azure/sql.tf# L50)
Query to find passwords and secrets in infrastructure code.

Passwords And Secrets - Generic Password
/instance.tf: [45](https://github.com/ACN-APPSAS/terragoat/blob/snyk-fix-f72334b67a62135581b29795d4e5b0e5//terraform/azure/instance.tf# L45)
Query to find passwords and secrets in infrastructure code.

Passwords And Secrets - Generic Password
/instance.tf: [11](https://github.com/ACN-APPSAS/terragoat/blob/snyk-fix-f72334b67a62135581b29795d4e5b0e5//terraform/azure/instance.tf# L11)
Query to find passwords and secrets in infrastructure code.

Sensitive Port Is Exposed To Entire Network
/ec2.tf: [83](https://github.com/ACN-APPSAS/terragoat/blob/snyk-fix-f72334b67a62135581b29795d4e5b0e5//terraform/aws/ec2.tf# L83)
A sensitive port, such as port 23 or port 110, is open for the whole network in either TCP or UDP protocol

Sensitive Port Is Exposed To Entire Network
/ec2.tf: [83](https://github.com/ACN-APPSAS/terragoat/blob/snyk-fix-f72334b67a62135581b29795d4e5b0e5//terraform/aws/ec2.tf# L83)
A sensitive port, such as port 23 or port 110, is open for the whole network in either TCP or UDP protocol

CVE-2023-26159
Npm-follow-redirects-1.14.6
Vulnerable Package

CVE-2023-44270
Npm-postcss-7.0.39
Vulnerable Package

CVE-2023-44270
Npm-postcss-8.3.11
Vulnerable Package

CVE-2024-25710
Maven-org.apache.commons:commons-compress-1.15
Vulnerable Package

CVE-2024-25710
Maven-org.apache.commons:commons-compress-1.20
Vulnerable Package

CVE-2024-28849
Npm-follow-redirects-1.14.6
Vulnerable Package

CVE-2024-28863
Npm-tar-4.4.19
Vulnerable Package

CVE-2024-28863
Npm-tar-6.1.11
Vulnerable Package

CVE-2024-29041
Npm-express-4.17.1
Vulnerable Package

CVE-2024-4067
Npm-micromatch-3.1.10
Vulnerable Package

CVE-2024-4067
Npm-micromatch-4.0.4
Vulnerable Package

Unpinned Actions Full Length Commit SHA
/build.yaml: [15](https://github.com/ACN-APPSAS/terragoat/blob/snyk-fix-f72334b67a62135581b29795d4e5b0e5//.github/workflows/build.yaml# L15)
Pinning an action to a full length commit SHA is currently the only way to use an action as an immutable release. Pinning to a particular SHA helps...

Unpinned Actions Full Length Commit SHA
/semgrep.yml: [14](https://github.com/ACN-APPSAS/terragoat/blob/snyk-fix-f72334b67a62135581b29795d4e5b0e5//.github/workflows/semgrep.yml# L14)
Pinning an action to a full length commit SHA is currently the only way to use an action as an immutable release. Pinning to a particular SHA helps...

Unpinned Actions Full Length Commit SHA
/pull_request.yaml: [14](https://github.com/ACN-APPSAS/terragoat/blob/snyk-fix-f72334b67a62135581b29795d4e5b0e5//.github/workflows/pull_request.yaml# L14)
Pinning an action to a full length commit SHA is currently the only way to use an action as an immutable release. Pinning to a particular SHA helps...


remote execution notifications/ logs?

Is there someway to continuously output information during long rsync sessions? Sometimes one is not sure if flightplan has stalled or the task is taking too long.
Thanks

running flightplan in debug mode should be more verbose (i.e. fly --debug).
Please note that this mode is not much more verbose than the standard mode, give it a try 👍


question: why use '.pb.rb' extension for generated classes?

i'm curious why beefcake generates ruby classes with an extension .pb.rb.
currently i rename the classes back to just .rb so they work with rails auto-loading. but curious why they're generated this way in the first place.

If I had to guess, it's to make it more obvious that the files are machine-generated, but this code predates my time on the project (/cc @streadway ).
For what it's worth, my main beefcake-using project has a rake task that concatenates the .pb.rb files into a single .rb file, and doesn't autoload.

thanks for the info. much appreciated.


Title value is not replaced

We have following two constructors in Link class:
public Link(string rel, string href, CuriesLink curie)
public Link(string rel, string href, string title = null)

I used following to create link:
  foreach (Priority p in theTask.PossiblePriorities)
  {
       Links.Add(LinkTemplates.TaskLinks.PriorityLink.CreateLink(new { taskID = theTask.TaskID,
                           priorityID = p.PriorityID, titleVal = "Test1" }));
  }

Following is my link template:
public static class LinkTemplates
    {

        public static class TaskLinks
        {
            public static Link TaskEntry { get { return new Link("self", "~/api/tasks/{taskID}"); } }
            public static Link PriorityLink
            {
                get
                {
                    Link l = new Link("priorities",
                                                "~/api/tasks/{taskID}/priorities/{priorityID}",
                                                "{titleVal}");
                    return l;
                }
            }
        }
    }

Following is the response.
{
  "TaskID": 1,
  "TaskName": "Task1",
  "CurrentPriority": "Medium",
  "Category": "IT",
  "_links": {
    "self": {
      "href": "/api/tasks/1"
    },
    "priorities": [
      {
        "href": "/api/tasks/1/priorities/101",
        "title": "{titleVal}"
      },
      {
        "href": "/api/tasks/1/priorities/103",
        "title": "{titleVal}"
      },
      {
        "href": "/api/tasks/1/priorities/104",
        "title": "{titleVal}"
      }
    ]
  }
}

It is not replacing the title value. What am I missing here?
Thanks.

Only the href uses template substitution. Title is just a simple string. Set title directly to "Test1" in your example instead of "{titleVal}".

It won't meet my requirement. I need to display name property of each Priority object. So, are you suggesting me to write one Link proerty for each priority by hand - like "public static Link PriorityLinkLow", "public static Link PriorityLinkHigh", etc. What if I need a new priority in my domain? I need to break my code - isn't it? Do you consider it as a good design, for a library to be used in projects?

The CreateLink method is merely a convenience for cloning a prototype and substituting URI template parameters into the Href of the newly cloned Link object.
To set the Title property, clone your prototype Link via CreateLink, then set the Title property, such as (modifying your code from above slightly):
foreach (Priority p in theTask.PossiblePriorities)
{
    var lnk = LinkTemplates.TaskLinks.PriorityLink.CreateLink(
        new { taskID = theTask.TaskID, priorityID = p.PriorityID});
    lnk.Title = whateverYouWantTheTitleValueToBe;
    Links.Add(lnk);
}


Ignore some routes

I'm in the process of migrating a project over from javalin 4 to 5. In the previous OpenAPI generator version, it was possible to exclude some routes from the generated JSON. We were using this to generate two different OpenAPI definitions where one would only include a subset of endpoints relevant for a specific type of client. Is it also possible to do this somehow in this version?

There's no such built-in feature, but you can always process stored openapi documentation using document processor:
https://github.com/javalin/javalin-openapi/blob/3e26ab1f4b54a720526e5cb59a6da6a027df25ec/examples/javalin-gradle-kotlin/src/main/java/io/javalin/openapi/plugin/test/JavalinTest.java#L128-L131
To e.g. remove/add entries, do you think it would be enough for your case, or do you have something else in mind?

Hey, thanks for the quick reply! I think the document processor should be sufficient for my needs.

Sure, let me know if you won't be able to achieve sane result with this approach, maybe we'll figure out alternative solution.

Fair point, the adjustment should be available in the 5.1.2-SNAPSHOT from maven.reposilite.com/snapshots repo when CI will finish all tasks.

Nice!

Just wanted to point out that there is "ignore=true" parameter in @OpenApi annotation which would exclude it from being generated. I tested it and works in my case.

Indeed there is. This option however would remove a path from any OpenAPI spec. In case you want to have multiple definitions, each only containing a subset of all available paths, you still need some kind of filter.


UFO splat left after new wave

"Other time got the red alien corrupt, but curiously after a while It fix alone".

Reset


SUBMARINE-172. [PoC] Split the core module to client and server module

What is this PR for?
PoC for the SUBMARINE-154 design. Design doc link: https://docs.google.com/document/d/1vCjt2z3DoagfeaVQV0nfyWqSYLymoP53rDcPcz-YcSc
What type of PR is it?
[Refactoring]
Todos
What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-172
How should this be tested?
https://travis-ci.org/jiwq/hadoop-submarine/builds/597733767
Screenshots (if appropriate)
Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

@jiwq , Please create new issue , fix modify the package names of tony runtime and yarn native service to org.apache.submarine.yarn.submitter.tony and org.apache.submarine.yarn.submitter.nativeservice respectively.

@jiwq , Please create new issue , fix modify the package names of tony runtime and yarn native service to org.apache.submarine.yarn.submitter.tony and org.apache.submarine.yarn.submitter.nativeservice respectively.

I had create the SUBMARINE-255 to track this and create the PR #57 to fix this.


Different browser versions between scripts

Hello,
I recently added browser-update through npm to a project, but I am currently having an issue with the browser versions in which the notification appears.
I'm using the following configuration and the notification is not displayed on Chrome 108 for example:
{
  required: {
    a: 114,
    c: 109,
    e: 112,
    f: 102,
    ios: 15.6,
    o: 98,
    o_a: 73,
    s: 15.6,
    samsung: 20,
    uc: 13.4
  },
  insecure: true,
  unsupported: false,
  api: 2023.07,
  text: '...',
  no_permanent_hide: true,
  reminder: 24,
  reminderClosed: 24
}

By looking at the code, I have found in update.npm.full.js the object vsakt containing browser versions. I also found in update.min.js the same object but with different browser versions.
Is there a reason for the differences of versions in the two scripts ?

You are right, this should be the same and be fixed now.

Thank you for resolving this issue. Are you considering upgrading the package version on npm?


TP1 loop

For some assets (i.e. ADA, RSR - not on BTC), the initial TP point gets in a loop (the order gets canceled and resubmitted over and over again, with the quantity adding up over at Bybit leading to a full 100% of the position on TP1).

Usually, the reason is not activating "Partial TP" on the exchange, and an old bug in previous version. Can you try now after adjusting the TP/SL option for the asset?


cmake: Fix support for non-standard psp source directory

Describe the contribution
This PR fixes a bug which prevents non-standard psp source directory specification via the environment variable $CFS_APP_PATH or the cmake variable ${psp_SEARCH_PATH}.
The cFE cmake build infrastructure uses the environment / cmake variable $CFS_APP_PATH
as well as the cmake meta-variable pattern ${${APP}_SEARCH_PATH} to locate apps and other modules (such as psp), and subsequently sets the variable pattern ${${APP}_MISSION_DIR} (see mission_build.cmake lines 202-222).
However, the resulting variable ${psp_MISSION_DIR} is not used in process_arch() in arch_build.cmake line 588 when including a target platform's build_options.cmake file. This causes a cmake configuration failure when trying to use a non-standard psp location.
Testing performed
Build with psp in non-standard location, with the CFS_APP_PATH environment variable set:
# set up standard cFS bundle per https://github.com/nasa/cFS
git clone --recursive https://github.com/nasa/cFS.git
cd cFS
cp -R cfe/cmake/sample_defs .
cp cfe/cmake/Makefile.sample Makefile
# hide psp
mkdir alt
mv psp alt/psp
# define alternate search path
export CFS_APP_PATH=$(pwd)/alt
# build sample mission
make prep
make

Before this PR:
...

-- Generated cfe_platform_cfg.h from /***REDACTED**/cFS/sample_defs configuration
CMake Error at cmake/arch_build.cmake:588 (include):
  include could not find load file:

    /***REDACTED***/cFS/psp/fsw/pc-linux/make/build_options.cmake
Call Stack (most recent call first):
  CMakeLists.txt:124 (process_arch)

-- Configuring incomplete, errors occurred!

...

After this PR:
...

-- Generated cfe_platform_cfg.h from /***REDACTED***/cFS/sample_defs configuration
-- BSP Selection: generic-linux at /***REDACTED***/cFS/osal/src/bsp/generic-linux
-- OSAL Compile Definitions: _LINUX_OS_;_XOPEN_SOURCE=600
-- OSAL Selection: posix at /***REDACTED**/cFS/osal/src/os/posix
-- PSP Selection: pc-linux

...

Built target mission-all

Expected behavior changes
This should only affect projects which use the environment variable $CFS_APP_PATH or the cmake variable ${psp_SEARCH_PATH}. Currently, they are being ignored for psp's build_options.cmake path resolution, which means some users may be building with different target build options than they think they are based on the intended behavior.
System(s) tested on

Hardware: PC
OS: Ubuntu 18.04
Versions: cFE v6.8.0-rc1+dev933

Additional context
Aside, this is also suggested as a method to specify a non-standard psp source directory in mission_defaults.cmake line 67 since PR #751 with the following:
set(psp_SEARCH_PATH ".")

However, this will lead to the same cmake configure error as above without this fix.
Third party code
No thirdy party code included in this contribution.
Contributor Info - All information REQUIRED for consideration of pull request
Jonathan Bohren, Honeybee Robotics

Thanks for the contribution @jbohren-hbr ! I opened a related issue to help us write the version description document. Can you double check it and add comments as needed?
https://github.com/nasa/cFE/issues/1921

#1921 looks good to me!

CCB:2021-09-01 APPROVED


Let the gcd package download its own datastore emulator.

Given that Travis and make test currently work properly, I think this is now a "nice to have" if we want the datastore/gcd code to be more of a reusable library.
However, it's a bit weird for a Go library to download a 100MB binary dependency (in Java) on first run.
I hope that we can test using a local emulator "properly" once https://github.com/GoogleCloudPlatform/gcloud-golang/issues/224#issuecomment-218327799 is implemented.


i2c: Adds a context object for i2c

Rather than a custom I2C_DEVICE_DT define macro use include
ordering and hooks into the original DEVICE_ macros to create a
common i2c context object used as a container type for the
device state.
This eliminates the need for any i2c driver to understand how to use
the context, initialize the context, or allocate the context.
It may be opted out of by not including i2c_context as there is
a magic word placed in memory to enable runtime checks, when needed,
if the driver has opted in to using the common code and data.
try it out with a frdm board if interested
west build -p always -b frdm_k64f samples/sensor/fxos8700 -- -DCONFIG_I2C_STATS=y -DCONFIG_STATS_NAMES=y -DCONFIG_SHELL=y -DCONFIG_STATS_SHELL=y -DCONFIG_STATS=y

Signed-off-by: Tom Burdick<EMAIL_ADDRESS>

While not passing CI, I'd like to know if people prefer this sort of approach over the device class macro prefix and copy/paste required there.

I like this idea. It's a bit magical, but in a consistent enough way that I think people will get used to it.


V3 (WIP)

@jaumard I'm not really sure how the original worked... When we pass in the handler as a string, it throws an express error.

@scott-wyatt all good for me :) take a final look and if it's ok we can merge/deploy this

Yeah, looks like this is still good, we will need to change the appveyor to only run tests on node 8 and above from now on since that is where Trails backwards compatibility is in v3


Could not set unknown property 'ideDirectory' for task ':runIde' of type org.jetbrains.intellij.tasks.RunIdeTask

I am trying out an Android Studio Plugin with the new IntelliJ Plugin SDK and ran into the following error when building the code using IntelliJ Ultimate Edition.
Build file '/Users/hle3/dev/plugin/build.gradle' line: 35

A problem occurred evaluating root project 'plugin'.
> Could not set unknown property 'ideDirectory' for task ':runIde' of type org.jetbrains.intellij.tasks.RunIdeTask.

* Try:
Run with --stacktrace option to get the stack trace.  Run with --scan to get full insights.

From the error, it looks like that ideDirectory is an unknown property, but according to the doc https://plugins.jetbrains.com/docs/intellij/android-studio.html#configuring-the-plugin-pluginxml-file, it is how you configure to run your plugin in android studio.
Here's my build.gradle file
plugins {
    id 'org.jetbrains.kotlin.jvm' version '1.5.10'
    id 'org.jetbrains.intellij' version '1.1.4'
}

group 'org.example'
version '1.0-SNAPSHOT'

repositories {
    mavenCentral()
}

dependencies {
    implementation 'org.jetbrains.kotlin:kotlin-stdlib:1.5.21'
}

compileKotlin {
    kotlinOptions.jvmTarget = "1.8"
}
compileTestKotlin {
    kotlinOptions.jvmTarget = "1.8"
}

// See https://github.com/JetBrains/gradle-intellij-plugin/
intellij {
    //version = '2021.2'
    // Define IntelliJ Platform against which to build the plugin project.
    version = '191.8026.42'  // Same IntelliJ IDEA version (2019.1.4) as target 3.5 Android Studio
    type = 'IC'              // Use IntelliJ IDEA CE because it's the basis of the IntelliJ Platform
    // Require the Android plugin, Gradle will match the plugin version to intellij.version
    plugins = ['android']
}

runIde {
    ideDirectory = '/Applications/Android Studio.app/Contents'
}

patchPluginXml {
    changeNotes = """
      Add change notes here.

      most HTML tags may be used"""
}

The runIde.ideDirectory property is renamed to runIde.ideDir since 1.0 version. Check the breaking changes guide.

thanks, @hsz  for the reply and updated documentation.
I've updated the code to use the ideDir.
runIde {
    ideDir = '/Applications/Android Studio.app/Contents'
    autoReloadPlugins = true
}

I am getting a different error this time, complaining that ideDir cannot take a string.  Could you provide an example of what ideDir path should look like to point to Android Studio?
I've looked at the README and the definitions of path are different in different places of the documentation.
FAILURE: Build failed with an exception.

* Where:
Build file '/Users/hle3/dev/plugin/build.gradle' line: 36

* What went wrong:
A problem occurred evaluating root project 'plugin'.
> Cannot set the value of task ':runIde' property 'ideDir' of type org.gradle.api.file.Directory using an instance of type java.lang.String.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 294ms

@HaiLe I think you should use ideDir.set(file("/Applications/Android Studio.app/Contents")) instead of ideDir = '/Applications/Android Studio.app/Contents'


[master]Add MTU suport for SLES and Ubuntu

Add code logic to parse MTU value from network info
and set to network configuration scripts for SLES and Ubuntu.
Signed-off-by: Shu Juan Zhang<EMAIL_ADDRESS>

@jichenjc @SeanHQF @bjhuangr Please help review. Thanks.
Not change RHCOS yet


Can't use Fossil commands

Trying to use commands from the fossil menu, like "fossil changes", "fossil extras" and "fossil diff" result in an error
I have fossil 2.17 installed and in my executable path.  The root directory of my project is the top directory of a fossil checkout.
I click on the F icon, the fossil menu opens, and I click on line 2 "fossil changes".  I get the error:
bad option "-fa": must be -ignorestderr, -keepnewline, or --
while executing
"exec -ignorestderr {}$args"
(procedure "::em::execcom" line 3)
invoked from within
"::em::execcom {}$term {}$lang {}$fs {}$geo {}$ttl {}$tpars  -e {}$composite"
(procedure "::em::xterm" line 14)
invoked from within
"::em::xterm $sel $amp"
(procedure "::em::term" line 3)
invoked from within
"::em::term $sel $amp"
(procedure "shell0" line 32)
invoked from within
"shell0 $sel $amp $silent"
(procedure "shell1" line 6)
invoked from within
"$c1 $typ "$seltd" $amp $silent"
(procedure "::em::Shell_Run" line 45)
invoked from within
"::em::Shell_Run $from $typ $c1 $s1 $amp $inpsel"
(procedure "shell_run" line 12)
invoked from within
"shell_run $from $typ shell1 $s1 $amp"
(procedure "shell" line 3)
invoked from within
"shell $typ $s1 $amp "button""
(procedure "::em::shell_button" line 2)
invoked from within
"{*}$comm"
(procedure "::em::pr_button" line 8)
invoked from within
"::em::pr_button 2 ::em::shell S: m2 &S"
invoked from within
".em.fr.win.fr2.butt invoke"
("uplevel" body line 1)
invoked from within
"uplevel #0 [list $w invoke]"
(procedure "tk::ButtonUp" line 22)
invoked from within
"tk::ButtonUp .em.fr.win.fr2.butt"
(command bound to event)

This issue is in my todo list for Windows. Hopefully it will be solved till 2022.

I should have mentioned that I'm using Linux.
The program looks like it will be very useful.  I hope you will be able to continue working on it.

Mea culpa. I overlooked the xterm of the error message.
I've changed the appropriate code. You can try it by the link:
https://chiselapp.com/user/aplsimple/repository/alited/zip/trunk/alited.zip
Also, there is a setting in "Setup/Preferences/Tools" named "Linux terminal" which can be reset, e.g. I use the following:
  lxterminal --geometry=220x55

that's very nice, though xterm is sort of standard.

I see now that the root of the problem is that it's not easy in all cases to choose a command to launch a terminal window.  I'm using Kubuntu where the terminal command is "konsole".  I found the following page helpful:
https://unix.stackexchange.com/questions/137782/launching-a-terminal-emulator-without-knowing-which-ones-are-installed
That link explains that "xdg-terminal" and "x-terminal-emulator" are commonly-used aliases for a terminal window command.
I put the value "x-terminal-emulator" in "Setup/Preferences/Tools/Linux terminal" and the fossil tools worked as expected.
Would it be practical to use tkcon as a final fallback?

Thank you for your kind advice. I'll put x-terminal-emulator into a writeable combobox of "Linux terminal" setting along with xterm and lxterminal.


WON'T MERGE: BP deployment

Description
Making a PR here to make it more visible.
Changes Made

Remove front-end from Dockerfile
Remove Metax
Remove Auth: use hardcoded user and project

btw, tests will fail :)

I wonder why there are commits from #453 here.

We shouldn't need this any more now that #583 and #548 have been merged.


⚠ Certificate extraction

What this PR does / why we need it:
This PR extracts certificate management from the Reconciler keeping the scope of the reconciler more focused.
It fixes #59 and fixes #217.
This is a BREAKING API change.
I would like to see this project move to a non-library layout as this code was not intended to used as a library and we are not versioning it as such.
EDIT: while writing this I realized it will fail CI due to godoc. Updates coming very soon.

How does all of this relate to https://github.com/kubernetes-sigs/cluster-api/tree/master/util/certs?

@vincepri I could use those and delete (a lot) more code!

/hold
need to run some local tests, will update with results

I wouldn't call the package internal (I'm trying to get us to move away from generic names like internal and util). You could do internal/certs?

@ncdc the logic behind the name was that it wasn't complex enough for its own package and certs.Certificates stutters a bit.
What about hmm, cluster?

Not that this is a valid justification, but there is at least context.Context as precedent.
I'm fine with internal/cluster for the time being, though!

/approve
Pending for @ncdc answer to comment before lgtm

@fabriziopandini over to you for lgtm (the f2f is keeping me busy) - thanks!

/lgtm


cosmosdb statestore partitioning broken under load

In what area(s)?

/area runtime

/area operator

/area placement

/area docs

/area test-and-release

What version of Dapr?

1.4.0

1.0.x
edge: output of git describe --dirty

Expected Behavior

Having only one db entry called actors||ACTOR_TYPE||metadata and n buckets, where n is the number of partitions.
Actual Behavior

There are multiple parent entries in db with name actors||ACTOR_TYPE for each id value, there are n partition-entries, in db containing 200 - 400 reminder entries. value property is set to null.
There is also one single actors||ACTOR_TYPE||metadata with also n partitions and a properly initialized value field.
Table shows parent entries in statestore. For each entry, there are also 50 partiotion-bucket entries as mentioned above.

id
partitionkey
ts
date
value-attribute

actors||ACTOR_TYPE
8e7c8966-ac47-41bb-8f98-1e4b4633feb3
1632737080
27.09.2021 10:04
null

actors||ACTOR_TYPE||metadata
actors||ACTOR_TYPE||metadata
1632737080
27.09.2021 10:04
{  "id": "8e7c8966-ac47-41bb-8f98-1e4b4633feb3",  "actorRemindersMetadata": {    "partitionCount": 50  }        }

actors||ACTOR_TYPE
6c8f8b84-d4bd-405e-a349-57065e644d73
1632735901
27.09.2021 09:45
null

actors||ACTOR_TYPE
a7e27bd7-dc5b-45cc-8760-60d615d3b455
1632734569
27.09.2021 09:22
null

The screenshots show the time frame in which the loadtest took place.
The loadtest started at 11:12:23am and created 2k actors each registering 7 reminders.
2k reminders where supposed to execute at 11:45. After registering those reminders, all actors have been deactivated before 11:45 and recreated with an execution tim at 11:35.
The reregistration started at 11:19:47am
actors||ACTOR_TYPE||metadata -> id == 8e7c8966-ac47-41bb-8f98-1e4b4633feb3

actors||ACTOR_TYPE -> id == 8e7c8966-ac47-41bb-8f98-1e4b4633feb3

actors||ACTOR_TYPE -> id == a7e27bd7-dc5b-45cc-8760-60d615d3b455

actors||ACTOR_TYPE -> id == 6c8f8b84-d4bd-405e-a349-57065e644d73

Steps to Reproduce the Problem

It was revealed only through our loadtest session. In normal load situation, error situation did not occur.
Release Note

RELEASE NOTE:

This seems to be a hot partition problem in CosmosDB. I recommend going through this documentation first: https://docs.microsoft.com/en-us/azure/cosmos-db/partitioning-overview#physical-partitions
Then, see if your provisioned throughput is enough to sustain this load. The throughput is evenly split across partitions, so there might be a need to overprovision to compensate for a hot partition.
Another strategy to minimize the hot partition problem is to play with the partition count. For example, try using a prime number. In your scenario, the next prime number is 53.

ok, thx. We will try the prime approach first.

The fact that you see Key "actors||ACTOR_TYPE" with PartitionKey "a7e27bd7-dc5b-45cc-8760-60d615d3b455" is odd because the first is the old way data was saved while the second is partitioned - which means the Id was supposed to be per reminder partition and not for the global "old" way.

I am looking into this issue.

To clarify, any instance of id "actors||ACTOR_TYPE" should have a partitionKey of "actors||ACTOR_TYPE" as well.

I reproduced a slightly different problem. The issue I found is that the "actors||ACTOR_TYPE" record is created even after the partition is set and none of the partition reminder records were used. I will find the root cause for this, which might lead to fix the issue you are seeing.

Hi @artursouza , sounds promising. As this is blocking us from moving forward with our releases, it would be great, to have a fix available soon.
Last loadtest run with 1009 partitions caused ~75.000 entries on the db, where only 1010 were expected due to this behavior ;-)

@javageek79 I found the root cause for the behavior I was seeing before. Let me explain my test scenario:
test client ---> dapr sidecar A -----> dapr sidecar B ---> actor app
The issue I was facing is because sidecar A was being invoked to register a reminder but it was not configured with the Actor.TypeMetadata feature. Only sidecar B had that configuration. Once I added this config, it worked as expected.
Can you confirm if the same is going on in your test scenario? Please, make sure that all sidecar instances have this preview feature enabled:
spec:
  features:
    - name: Actor.TypeMetadata
      enabled: true

Hi @artursouza, the feature is enabled in a config.yml which is referenced only by sidecar B.
The connection between sidecar A and sidecar B is via Azure Service Bus. Therefore have no direct connection.
Does every dapr related service needs to reference the config to make the feature work?
How can I verify that the feature was enabled in sidecars?

Moving this to 1.6. The action item is to remove the transaction logic and use batch write instead since transaction is not really required. Also, we will change the partitionKey to be simply the key - which is a breaking change and will need to provide a script to migrate the data.

We have changed how reminder partitions are migrated in 1.6 by using batch instead of transaction, migrating only at start up and not on registering/unregistering reminders and blocking multiple migrations in parallel from the same process. This should reduce contention in reminder persistence.
In addition, we also fixed a bug that was allowing multiple state stores for actors to be randomly selected instead of failing the sidecar right away.
Please, observe these in 1.6 release to confirm if issues continues.


KOGITO-8229: Dashbuilder ECharts Native

Bringing ECharts as the default chart library for the following types:
BAR, LINE, AREA, BUBBLE, PIE and METRIC.
C3 will remains as an alternative
The main reason for this change is that ECharts has a better native support for dark mode and easier to customize
For this reason the following attributes are now supported:
general.mode: dark or light;
extraConfiguration: Additional JSON configuration based on ECharts Option object. This configuration overrides the configuration used by Dashbuilder and allow users to build more complex visualization. The only limitation with this feature is that the extra configuration is JSON, not a Javascript object, so using callbacks on formatter does not work.
https://user-images.githubusercontent.com/359820/201425099-9f5f365f-289f-4c20-a74d-8b785676dc1a.mp4
A new chart type is also now supported. SCATTERCHART is a new type and it was "for free" on echarts, so we now support it.
A Basic echarts usage is as following. Notice it is in dark mode and contains a title
datasets:
- uuid: test
  content: >-
            [
                ["A", 1, 3],
                ["B", 2, 5],
                ["C", 3, 1]
            ]
pages:
- components:
    - settings:
        type: BARCHART
        general:
            title: Simple Bar Chart
            mode: dark
        lookup:
          uuid: test

You can change to c3, but dark mode will be ignored. Just add a field called "renderer" with the value "c3"
datasets:
- uuid: test
  content: >-
            [
                ["A", 1, 3],
                ["B", 2, 5],
                ["C", 3, 1]
            ]
pages:
- components:
    - settings:
        type: BARCHART
        renderer: c3
        general:
            title: Simple Bar Chart
            mode: dark
        lookup:
          uuid: test

Which renders to the following:

Finally, some advantages of using echarts we added new properties:

export.png: adds an icon to download the PNG of a chart
general.allowEdit: an icon when clicked open the data in text mode and allow users to edit the data used by the chart
bubble.minSize: a min size for bubble chart
bubble.maxSize: a min size for bubble chart

bubble.color: Select the bubble color

To run this build until "dashbuilder-editor" and run the project as a web application. This is already live on my test editor: https://jesuino.github.io/dashbuilder-client/

https://issues.redhat.com/browse/KOGITO-8252
Since the very reason for this PR is dark mode, so this PR also includes dark mode. To achieve this new feature we added support to a new top level attribute called "global" and it supports two fields:

global.mode: Can be dark or light - it is the mode for the entire report;
global.settings: It is a global settings object. It has a global setting that can be applied to all components on the dashboard, except for the lookup, the lookup is not a global configuration. The  component declared in the layout can override the global setting

Hello @caponetto
Thanks for your review, updated the code with your suggestions

Hey @jesuino, the code looks good to me but there are some failing integrations tests.
Can you please check? I think the test is trying to get C3 stuff where it doesn't exist anymore.

Hi @jesuino,
is this the final version with all the commits? I did some basic manual check of the PR last week and it did look good, but I want to take a final look at the UI with all the commits before it gets merged.

Hello @barboras7
It is the final version now - I added dark mode on top of it because we would double our work to review another PR
If you see new commits it is just because I am testing the dark mode and sometimes I find small bugs
Thanks!

Just for information
cypress is failing with a weird error due this: https://github.com/cypress-io/cypress/issues/24652
I updated cypress to 1.10.0 and it worked

Hello @barboras7
Thank you so much for your review.

I like the dark mode, however if you google something like "charts dark mode", you don't only get specific chart in the dark mode but everything rendered is in dark. I understand that this would require much more work, but maybe something to think about for future. It looks a bit strange to just to have one dark chart with everything white around it.

The local declaration of mode is override by the global mode., see this example:
global:
  mode: dark
datasets:
  - uuid: test
    content: >-
                [
                    ["A", 1],
                    ["B", 2],
                ]
pages:
  - components:
      - settings:
          type: BARCHART
          lookup:
            uuid: test

This renders to the following

You can see a full example here:
https://jesuino.github.io/dashbuilder-client/?import=https://raw.githubusercontent.com/jesuino/dashbuilder-yaml-samples/main/random/fifa_2022_goals.dash.yaml#editor=false&preview=true


Brian moir

Here is my submission for my Sprint. I feel like i did much better than my earlier project this week.

brian you didn't fork the repo


Q. Simulation Options

Hello,
The simulation code includes five different options, but their specific purposes and functionalities are not clear.
Could you please explain what each of these options corresponds to and how they affect the simulation?
Having a detailed understanding of these options would be very helpful for running and reproducing the experiments accurately.
Thank you.

Thank you for your question.
The simulation code's options (op1-op5) correspond directly to the five items listed in Figure 12 of the paper. Below is a brief explanation for each:

Option 1: Cluster-Alloc
Option 2: Node-Scaling
Option 3: GPU-Scaling
Option 4: Clock-scaling
Option 5: Baseline

If you need further clarification on the other options or additional details about how these relate to the simulation, feel free to ask.
Thank you!


available_terminals methods has moved to Settings, error should reflect it

https://github.com/dilcom/pilot-gnuplot/blob/555e6886e4ed32053fe97937327c92c5a8535d65/lib/pilot-gnuplot/mixins/option_handling.rb#L66

Sure, will fix


Lock capnproto to v1.0.0

Lock capnproto to v1.0.0

is this needed ? I found that capnproto even down to 0.7 works just fine.
Requiring vert specific versions makes things hard to package.

The reason to lock is that the tip now needs gcc 11 or higher because of C++ 20 requirements. Surelog is still on gcc 9. Capnproto v1.0 is the last good release which still works with gcc 9.

Sounds good.


withdirs param not working w/ find

Hi!
Referencing the find function of abstract file system, it has a param withdirs. Although irrespective of the value of withdirs, the dropboxdrivefs find function always seems to return a list including directories.
Just documenting the issue for now, no immediate fix is required.

Dear, sorry for the delay.
Looks like it is fixed now. I added the example in the notebook.
If you see I misunderstood or you still have the issue on your side, please tell me which package/version of fsspec you are using in background.
I will close it for now. Thank you for your reporting.


chore(master): release 0.42.0

:robot: I have created a release beep boop
0.42.0 (2024-06-04)
Features

support add document to multiple context paths (#412), closes #407 (0107676)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/easyops-cn/docusaurus-search-local/releases/tag/v0.42.0 :sunflower:


Add a map showing all the countries that fellows are located in

Fixes #10
Please, go through these steps before you submit a PR.

[x] My Pod Leader knows I'm working on this Pull Request
[x] I've explained what the Pull Request is adding.
[x] I've explained why this is important.

Description

Added an interactive map using LeafletJS
Added circular profile pictures to show where everyone is on the map

Hey @vulongphan. Can you merge this pull request?

Hey @vulongphan, can you merge this pull request?


Static legend handlers

And a few small changes to the camera plotting, the only important one is a sign change for the rotation of the pixels which was not propagated before.

Codecov Report

Merging #347 (479d5b6) into master (aa30187) will increase coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #347      +/-   ##
==========================================
+ Coverage   76.96%   76.98%   +0.02%
==========================================
  Files          67       67
  Lines        6454     6461       +7
==========================================
+ Hits         4967     4974       +7
  Misses       1487     1487

Impacted Files
Coverage Δ

simtools/model/camera.py
39.85% <100.00%> (ø)

simtools/util/legend_handlers.py
50.00% <100.00%> (+4.02%)
:arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Rationale for abandoning mutable references

Hello! I just came across https://github.com/godot-rust/godot-rust/issues/301, which bans &mut self from generated methods and uses &self even for mutating operations.
Concrete example -- the BoxShape struct is now generated as follows (omitting irrelevant parts):
pub struct BoxShape {
    this: object::RawObject<Self>,
}
impl BoxShape {
    pub fn new() -> object::Ref<BoxShape> { ... }
    pub fn extents(&self) -> Vector3 { ... }
    pub fn set_extents(&self, extents: Vector3) { ... } // shared-ref!
}

To me, it seems that I lose a lot of type safety, as I could no longer guarantee that:
// signature implies immutable
fn compute_volume(shape: &BoxShape) -> f32 { ... }

would not change its argument by accident.
A central part of the type system -- mutability -- is now disabled. This will probably be surprising to users, and could even be one of the reasons why they choose a strongly typed language like Rust instead of GDScript in the first place.
In the C++ binding, const correctness is followed as expected (but there, mutability isn't related to aliasing).

In the issue, the decision is justified as follows:

The shared-mutable distinction in generated types cause inconveniences in codebases, like seen in the from_sys(to_sys()) "idiom". It also prevents aliasing from being a safe action, which is desirable. It should be removed since Godot types always allow mutable aliasing internally.

As I (and probably others) haven't been around to see such "bad idioms" emerge, would it be possible to elaborate the reasoning, and maybe even include it in the gdnative's main doc page?
In particular, why not the "standard" way of using RefCell etc.? (ergonomics?)
[Edit] now that I think about it, can godot-rust maybe not guarantee that a mutable reference has exclusive access, and the consequences of that broken promise is worse than losing type safety?

Unrelated, I also found the explanation in https://github.com/godot-rust/godot-rust/issues/390 very helpful, is that already part of the docs somewhere? If not, a place where such semantics are highlighted would be very appreciated!

Hey @Bromeon! This is all still pretty fresh in the pipe, but a short summary is:

Godot objects can be accessed and aliased freely but Rust's &mut expects only a single mutable alias at a time. In terms of type "safety" this was not making anything more or less secure. I do agree that it made methods more "self documenting" as it was expressing whether a function could mutate or not.
It makes a lot of sense if you view & and &mut not as "immutable" and "mutable" but as "shared" and "unique". The Rust bindings can't guarantee that access will be unique, so & is more a more accurate modelling.
The reason no thing such as RefCell is used because the types already have interior mutability from Rust's point of view. Adding an actual Rust RefCell would only track usage from Rust, add overhead and would not work for any access from GDScript, AnimationPlayers or other types of scripting.
The docs about this are not very complete as this change of reference types happened within the last week and we are still making a lot of changes. There is a tracker for a documentation book in #330 which could probably benefit from a more detailed in-depth explanation.

Does this clear things up a little?

Hey, thanks a lot for the answer!

It makes a lot of sense if you view & and &mut not as "immutable" and "mutable" but as "shared" and "unique".

Ah, ok! That was my suspicion (see edit), but this sentence words it perfectly.
Godot aliases objects in the sense that multiple locations can store (non-const) pointers, like it is always done in the SceneTree -- but not to allow concurrent modifications (apart from thread-safe APIs), right?
As such, is aliasing a T& from multiple threads not a safety hazard? Or does the godot-rust API enforce user's unsafe opt-in at thread boundaries, to make sure he's aware of the consequences?

The reason no thing such as RefCell is used because the types already have interior mutability from Rust's point of view. Adding an actual Rust RefCell would only track usage from Rust, add overhead and would not work for any access from GDScript, AnimationPlayers or other types of scripting.

Makes sense. As far as I understand, we have to get as close as possible to C++ T*/T& or GDScript references, which are both mutable and aliased, and in Rust have the following trade-offs:

Rust &mut T -- mutable, but not aliased
Rust *mut T -- mutable, aliased, but unsafe
Rust RefCell<T> -- immutable, aliased, but verbose

So the decision was to have kind of an "intrusive" RefCell within each object and thus avoid the syntax overhead.
Just out of curiosity, did you ever consider immutable views, as they are common in Java, C# etc? That is, you would have two types BoxShape and ImmutableBoxShape, where the latter only offers the immutable subset of methods. It would likely bloat the API too much and is not very idiomatic in Rust, but just came to my mind, as I thought about the different languages' way to deal with this problem.

The docs about this are not very complete as this change of reference types happened within the last week and we are still making a lot of changes. There is a tracker for a documentation book in #330 which could probably benefit from a more detailed in-depth explanation.

Cool, I'll have a look at it. I'll probably have to play around with all the things first 🙂

As such, is aliasing a T& from multiple threads not a safety hazard?

Rust prevents that statically: &T is only Send (can be sent to another thread) if T is Sync (can be shared from multiple threads), and all Godot API objects are !Sync, which means that you cannot alias a &T and send it to another thread. If you try to do that, the compiler will complain about it. You can send a Ptr<T>, which is semantically a raw pointer, but it is unsafe to use, so no safety is lost here.

So the decision was to have kind of an "intrusive" RefCell within each object and thus avoid the syntax overhead.

No. It's because semantically all the Godot objects already include a RefCell.

Rust RefCell<T> -- immutable, aliased, but verbose

RefCell is a runtime borrow checker -- it's not a smart pointer and you can't really compare it with reference types. Adding RefCells in the bindings only adds runtime cost and has zero effect on safety.

Just out of curiosity, did you ever consider immutable views, as they are common in Java, C# etc?

We have the typestate pattern implemented for VariantArray and Dictionary, which is like this but better (as in being more general). However, here it isn't necessary because it is OK to mutate the objects as long as it is valid and the thread safety guidelines are observed in non-Rust code.

Also, see this essay by dtolnay about "mutable" references in Rust if you're confused, he explains it better than I can: https://docs.rs/dtolnay*/dtolnay/macro._02__reference_types.html

Thanks a lot for the detailed explanation, and the great article!

No. It's because semantically all the Godot objects already include a RefCell.

I worded it unfortunately; what I had in mind was something like C++'s boost::intrusive_ptr, where the pointee object would already provide the semantics for reference counting (and here, the semantics for interior mutability). But you're right to highlight that it shouldn't be directly compared to a smart pointer (let alone to C++).

We have the typestate pattern implemented for VariantArray and Dictionary (on master), which is like this but better (as in being more general). However, here it isn't necessary because it is OK to mutate the objects as long as it is valid and the thread safety guidelines are observed in non-Rust code.

Ah, nice. It looks like you differentiate the type-state based on shared vs. unique, and allow structure-mutating operations only in the unique case. Element-mutating operations are probably not considered dangerous, as they cannot invalidate the collection.

Closing because the question is answered.


Add blog posts about new sponsors

Want to add some puppet at CERN blurb. How we use puppet at CERN?


Update docs

Some significant update to

docs/syntax/index.html
docs/syntax/dynamic.html

Streamlining documents under

docs/functions
(preparing them for being navigational

There are a few issues to check

[ ] default property seems to have an issue. See ticket #188. Is it the program, or the documentation that needs fixing?
[ ] sel.path.title has been there twice (Line 16 and Line 26). Assuming it can be safely deleted as they have identical descriptions. Was there another property supposed to be there?
[ ] sel.raw was there twice, too. In the beginning of the document and towards the end. These are unlikely to be the same, since the later is requiring a parameter. What is correct here?
[ ] sys.name (just one after the other) and sys.type (also there) are duplicates. Save to be removed, I guess?
[ ] Ticket #189 prevents merging. If the server can not be adapted, the docs need to be updated.
[ ] Some inconsistencies with code vs monospace are to be expected. Without the valid CSS files, it is difficult to review the layout
[ ] The import tag is already documented according to #185. So this might need some tweaking.

Happy for feedback.

Awesome. Does not look too bad. I'll have a closer look and another go later on.


ADD iostream in CPP cheatsheet

PLease add my PR

PLEASE ACCEPT my PR!

Hello @N-E-W-T-O-N
Thanks for your contribution.
Let's connect ->
GitHub -> @Suryansh1720001 (Follow)
LinkedIn -> https://www.linkedin.com/in/suryanshprajapati17
Also, Star the repo ⭐


nxp,mcux-usbd devicetree binding issues

Describe the bug
The point of a  devicetree binding is to describe hardware and its initial configuration. We have carved out a few exceptions for driver-specific property names prefixed with zephyr,. However, we should not be designing devicetree bindings to cater to the software requirements of vendor HALs.
This USB device driver binding has a name that seems to specifically refer to the HAL itself, not the actual hardware. It also has problematic properties, such as usb-controller-index, added in 3a7c719fb45ea33f35a6464c8bbaa736ab6fc8cb.
Expected behavior
Devicetree bindings that reflect the hardware, not the software. Removal of this binding and replacement with individual bindings for the IP blocks in question, as many as needed. Removal of HAL-specific configuration properties.

@mbolivar-nordic. I am looking to address this change as part of an updated MCUX USB driver that would use the new USB framework.


Improve multiline description detection

As we have expanded our use of this gem we've found even after recent changes, heredoc description support is patchy. This PR attempts to DRY up the implementation for that, and improve it, with tests to boot.

Awesome, thank you!


Some notes about the new paradigm

Hey Folke,
I was thinking, to avoid or redirect a lot of similar questions regarding "spam" and/or issues with other plugins, there could be some explanation about the message system and a list of known conflicting plugins
Maybe tomorrow I can write up a small post in the discussions about :messages, shortmess, vim.notify etc that send or control notifications to various places that are now being unified and popped up by noice.
Several of these might be what people already know but pay no attention to as they show up in the message area when cmdheight is not 0. But now as noice pops them up, users are alarmed.
Maybe you could link it in the readme once I post it?

That would be great!
The Wiki might also be a good place for that info.

Will close this issue to not take space from relevant stuff.
Will write something up tomorrow

Awesome, thanks!

I have a draft ready.
Cant for the life of me figure out how to send a PR to wiki
So, I have just cloned the wiki, changed the remote and created branches+commits.
Will figure out what to do later today.

I've never done that before myself, so no idea :)
Thank you for documenting this!
I've maed some changes to health checks earlier today.
When another plugin overrode something, we'll now show a message that mentions the plugin name, file and line. And also how to disable the functionality in Noice

@folke
Here is the link to the first draft
https://github.com/ranjithshegde/noice.nvim/wiki/A-Guide-to-Messages
I found this answer on stack that explains what the owner of the repo needs to do to merge it
https://stackoverflow.com/a/56810747

Looks really great!
Some notes:

Your fork doesn't seem to have the original pages, so not sure how I can eventually merge that.
For showmode, maybe also add the statusline component? https://github.com/folke/noice.nvim/wiki/Configuration-Recipes#show-recording-messages
the section on LSP progress is not ideal, since that code snippet will override the Noice listener, so should be made very clear that they need to disable config.lsp.progress for that. Also worth mentioning that diagnostics config for update_on_insert. from that earlier issue.

Thanks again!

For some reason the forks dont pull in the wiki. Cant seem to understand how.
Will add showmode notes
Agree, it does override. I will make that explicit, and also add the diagnostics note

This link might be of interest
https://gist.github.com/larrybotha/10650410

I think I was able to merge it all, but had some conflicts in the other wiki pages. Not sure I manged to get everything there.
Thanks again for doing this!

I believe you need to


Tooltip on scroll container

Heys! Cool lib! while scanning through the demos spotted a small bug on tooltip positioning.
Calculation is wrong when tooltip on scroll container, because parent.offsetWidth < parent.scrollWidth.
demo
Possible Solution (pseudo-code)
Since it's not parent that is scrollable, but some outer wrapper div you could:

Add class targeting the scroller ('frappe-chart-outer') or something
Check if scroller is scrollable this could be done with

scrollWidth > offsetwith ? isScrollable : notScrollable

then use which suits best on calc_position()

this.scroller = 'scrollerEl';
calc_position() {
  ...
  let max_left = (isScrollable? this.scroller.scrollWidth : this.parent.offsetWidth)  - this.container.offsetWidth;
}

As i've stated, this is pseudo code and i haven't tested it. I might try it and submit a PR if you want but probably only have time on Friday.
Cheers and good luck with this!

Hi, send in a PR.

Thanks for reporting! We'll try and fix this asap, but a PR would be wonderful :D


PI-1115 update address search - again

What did we need to change?  I thought we already matched the AC and PI-1115 just needed us to check it / provide evidence to NDST?


CPI 'create_stemcell' method responded with error: CmdError{"type":"Bosh::AzureCloud::AzureError","message":"get_token - http code: 400. Azure authentication failed: Bad request

creating stemcell (bosh-azure-hyperv-ubuntu-xenial-go_agent 97.12):
CPI 'create_stemcell' method responded with error: CmdError{"type":"Bosh::AzureCloud::AzureError","message":"get_token - http code: 400. Azure authentication failed: Bad request. Please assure no typo in values of tenant_id, client_id or client_secret/certificate. Error message: {"error":"unauthorized_client","error_description":"AADSTS700016: Application with identifier 'id1' was not found in the directory 'id2'. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You may have sent your authentication request to the wrong tenant.\r\nTrace ID: c3cde515-223c-4620-90fd-46799e\r\nCorrelation ID: 493270da-492d-4bc1-8d1e-\r\nTimestamp: 2019-03-31 13:59:13Z","error_codes":[700016],"timestamp":"2019-03-31 13:59:13Z","trace_id":"c3cde515-223c-4620-90fd-46799e9","correlation_id":"493270da-492d-4bc1-8d1e-4c0aa45e","error_uri":"https://login.microsoftonline.com/error?code=700016\"}","ok_to_retry":false}
Exit code 1
Versions (please add them if you have more releases for this issue)
CPI      Version:
BOSH     Version:
CF       Version:
Diego    Version:
Stemcell
Version:
Distro:

[x] Ubuntu
[ ] CentOS

Environment

Azure Environment:

[ ] Azure
[ ] AzureChinaCloud
[ ] AzureUSGovernment
[ ] AzureGermanCloud
[ ] AzureStack

Version: (e.g. 1712 update or build number 180106.1)

[ ] Others:

Location: (e.g. East US)
Subscription ID:
Resource Group Name:
[ ] Use Managed Disks

Issue Details
Logs (Please redact all sensitive information in your logs)
Repro steps (if necessary, delete otherwise)

@ramintagizade
Can you double-check whether the application with identifier 'id1' exists in the directory 'id2'?
if the application exists, can you login azure cli/powershell with your service principal?

I solved that issue, by replacing with a new PCI, but now below error occurs , while wanting to grant a Contributor role to a new one: >az role assignment create --assignee "d00fc3c0-8b1c-4308-914e-ab" --role "Contributor" --scope /subscriptions/ad092f72-8200-4fb9-99de-0e
The client<EMAIL_ADDRESS>with object id 'g6ab4c39-05f0-42ed-a4ed-03a5c1' does not have authorization to perform action 'Microsoft.Authorization/roleAssignments/write' over scope '/subscriptions/ad092f72-8200-4fb9-99de-0e'.

This error occurs while i am starting bosh deployment on azure vm :
creating stemcell (bosh-azure-hyperv-ubuntu-xenial-go_agent 97.12):
CPI 'create_stemcell' method responded with error: CmdError{"type":"Bosh::AzureCloud::AzureError","message":"http_get - http code: 403. Error message: {"error":{"code":"AuthorizationFailed","message":"The client '4755fa65-6bbc-47a0-ae84-3354ada69' with object id '4755fa65-6bbc-47a0-ae84-3354ada69' does not have authorization to perform action 'Microsoft.Storage/storageAccounts/read' over scope '/subscriptions/ad092f72-8200-4fb9-99de-0e/resourceGroups/modb-chrt/providers/Microsoft.Storage'."}}","ok_to_retry":false}
Exit code 1

I solved that issue, by replacing with a new PCI, but now below error occurs , while wanting to grant a Contributor role to a new one: >az role assignment create --assignee "d00fc3c0-8b1c-4308-914e-ab" --role "Contributor" --scope /subscriptions/ad092f72-8200-4fb9-99de-0e
The client<EMAIL_ADDRESS>with object id 'g6ab4c39-05f0-42ed-a4ed-03a5c1' does not have authorization to perform action 'Microsoft.Authorization/roleAssignments/write' over scope '/subscriptions/ad092f72-8200-4fb9-99de-0e'.

The client<EMAIL_ADDRESS>doesn't have sufficient privileges to create role assignments. See: https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal#prerequisites

In my subscription i have a Contributor role, may be i need full admin access to write ?


Plugin failing to load

I installed the plugin using the instructions. When I tried to serve the site, I got this error. Any ideas?
$ hexo server
ERROR Plugin load failed: hexo-generator-json-content
SyntaxError: Block-scoped declarations (let, const, function, class) not yet sup               ported outside strict mode
    at Object.exports.runInThisContext (vm.js:53:16)
    at C:\hexo\manjaro-rocks\node_modules\hexo\lib\hexo\index.js:227:17
    at tryCatcher (C:\hexo\manjaro-rocks\node_modules\hexo\node_modules\bluebird               \js\release\util.js:16:23)
    at Promise._settlePromiseFromHandler (C:\hexo\manjaro-rocks\node_modules\hex               o\node_modules\bluebird\js\release\promise.js:510:31)
    at Promise._settlePromise (C:\hexo\manjaro-rocks\node_modules\hexo\node_modu               les\bluebird\js\release\promise.js:567:18)
    at Promise._settlePromise0 (C:\hexo\manjaro-rocks\node_modules\hexo\node_mod               ules\bluebird\js\release\promise.js:612:10)
    at Promise._settlePromises (C:\hexo\manjaro-rocks\node_modules\hexo\node_mod               ules\bluebird\js\release\promise.js:691:18)
    at Promise._fulfill (C:\hexo\manjaro-rocks\node_modules\hexo\node_modules\bl               uebird\js\release\promise.js:636:18)
    at Promise._resolveCallback (C:\hexo\manjaro-rocks\node_modules\hexo\node_mo               dules\bluebird\js\release\promise.js:431:57)
    at Promise._settlePromiseFromHandler (C:\hexo\manjaro-rocks\node_modules\hex               o\node_modules\bluebird\js\release\promise.js:522:17)
    at Promise._settlePromise (C:\hexo\manjaro-rocks\node_modules\hexo\node_modu               les\bluebird\js\release\promise.js:567:18)
    at Promise._settlePromise0 (C:\hexo\manjaro-rocks\node_modules\hexo\node_mod               ules\bluebird\js\release\promise.js:612:10)
    at Promise._settlePromises (C:\hexo\manjaro-rocks\node_modules\hexo\node_mod               ules\bluebird\js\release\promise.js:691:18)
    at Promise._fulfill (C:\hexo\manjaro-rocks\node_modules\hexo\node_modules\bl               uebird\js\release\promise.js:636:18)
    at C:\hexo\manjaro-rocks\node_modules\hexo\node_modules\bluebird\js\release\               nodeback.js:42:21
    at C:\hexo\manjaro-rocks\node_modules\hexo\node_modules\hexo-fs\node_modules               \graceful-fs\graceful-fs.js:78:16
    at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:380:3)
FATAL Port 4000 has been used. Try other port instead.
FATAL Something's wrong. Maybe you can find the solution here: http://hexo.io/do               cs/troubleshooting.html
Error: listen EADDRINUSE <IP_ADDRESS>:4000
    at Object.exports._errnoException (util.js:870:11)
    at exports._exceptionWithHostPort (util.js:893:20)
    at Server._listen2 (net.js:1231:14)
    at listen (net.js:1267:10)
    at net.js:1376:9
    at nextTickCallbackWith3Args (node.js:453:9)
    at process._tickCallback (node.js:359:17)
FATAL listen EADDRINUSE <IP_ADDRESS>:4000
Error: listen EADDRINUSE <IP_ADDRESS>:4000
    at Object.exports._errnoException (util.js:870:11)
    at exports._exceptionWithHostPort (util.js:893:20)
    at Server._listen2 (net.js:1231:14)
    at listen (net.js:1267:10)
    at net.js:1376:9
    at nextTickCallbackWith3Args (node.js:453:9)
    at process._tickCallback (node.js:359:17)

Sorry for late response!
Did you already fix it?
You can see the problem at line started with "SyntaxError".
Latest version was coded with ES6 syntax. You need latest version of Node.js to compile it correctly.

After you contacted me, I checked what versions I had of nodejs and npm. npm was at version 2.14.20 and node was 4.4.0. I updated both and now have npm 4.0.5 and node 6.9.1. I haven't had a chance to see if it fixed the problem. Will reply when I try it.

Ok, I'll wait for your feedback.

Hi, when I npm install -S  hexo-generator-json-content

And then I hexo s ,I got en error like @JohnBlood ,How can I solve this?

Ok, I'm waiting for your feedback.

@alexbruno I finally got a chance to try it. I reinstalled the plugin and ran hexo server. I didn't get an error and the search works nicely. Thanks for the advice.


Git URL's as Dependencies

Hello there!
Thank your for this project.
This PR brings jam ability to use git url's as dependencies (like in npm) in package.json file.
For example:
// package.json

{
  "jam": {
    "dependencies": {
      "dm": "git://github.com/gobwas/dm.js#0.3.0"
    }
  }
}

Available formats are listed in README.md. Also, as like in npm.
Thanks!

Hi, @caolan!
Thank you for your library.
If you do not want to merge this PR by any reason, we could create a new package, that based on fork of yours.
In our company we need dependencies resolution in flat manner, so Jam is being pretty good for this job. But we also want to use git dependencies, and some other features, like not only require.js config generation, but some other bundlers too.
Do you have any objections/restrictions on this?
Thanks!


Standalone Power Adapter Trigger - No Option for Unplugged

The power adapter trigger (battery level slider all the way to the right) does not auto-disable until the battery drops to 99%, rather than disabling as soon as power adapter is unplugged. May be as simple a fix as <=100 instead of <100 in the code.
Note: Label actually reads "Ask to end sessions if battery % is below..." The behavior described appears consistent with the function. Should instead consider a slider value for "Unplugged" at far right.

Created an option to the far right on the slider for "not charging." This will allow for a prompt as soon as the power adapter is unplugged.


xxhash + zstd

Maybe it would be worthwhile to add compression with "zstd" and to replace "crc32" with "xxhash"?

crc32c was chosen because intel provides primitives that can be used with 1cycle/byte efficiency for larger values.  I'd rather stick with the well known in this case.
As for zstd, compression would be best implemented at the block level of the key-value store, like what LevelDB's snappy does.  It is possible to add snappy support with a line or two to the key-value store.
LevelDB is not a long term choice of backend, so I have not taken the time to pull out every trick in the book.
In any case, 99% of the costs of a transaction come from network latency and not the local CPU usage.


fix: avoid parsing old value if not exists

Please refer: https://github.com/PlasmoHQ/storage/issues/6

Hey, @prakashchokalingam thanks for the pull request, and for helping us fix this bug!
I think this fix can be condensed to one line:
oldValue: JSON.parse(changes[key].oldValue || null)

JSON.parse doesn't know what undefined means but does know what null means as it's a valid JSON identifier.

Hi,  @ColdSauce  Thought the same. Yet JSON.parse(null) will add the parse computation, Hence skipping the JSON parse itself.
I am fine with both approaches.

Hmm I do think that keeping it undefined (i.e, when it is not initialized) would be cleaner. undefined is a runtime concept vs null is an user concept. If something is not yet set, it should stay undefined

Thanks @ColdSauce @louisgv


Support for Ivy

We should investigate what would be the changes required to support Ivy.

@wKoza I would love to support here as well :-) So how are your investigation so far?


#215 Improve proxy handling

If we are in a hosted enivornment (portal or AGOL), the proxy will
automatically be set to the hosted sharing proxy.  No matter where the
application is deployed, the JSAPI proxy url will now be set to the
appropriate value.
Fixes #215

@MikeTschudi Could you review this, please?

@allisonmuise, we can't yet use this improvement:

Rather than merge the pull request, we should merge branch improve-hosted-proxy-handling, which includes both this pull request and one small addition. I just updated the branch using the current develop branch.

The app fixes mentioned in https://github.com/Esri/map-and-app-gallery-template/issues/215#issuecomment-297172235 need to be done before this work can be released. The app continues to include query parameters in the URL that it sends to esriRequest rather than using the content parameter, so we're likely to get malformed URLs again.

closing this PR as this app has been moved to mature support and no further dev work will be done. Please use the Minimal Gallery template to create gallery applications.


Test client TLS

Codecov Report

Merging #26 (2b807f9) into master (5b9c2b8) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #26   +/-   ##
=======================================
  Coverage   64.33%   64.33%
=======================================
  Files           8        8
  Lines         314      314
=======================================
  Hits          202      202
  Misses         93       93
  Partials       19       19

Flag
Coverage Δ

unittests
64.33% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5b9c2b8...2b807f9. Read the comment docs.


Added Chihuahua starter

Hey @bgodlin, thanks for the review. I tried to reference all the source files that I've used. The commented out functions exist in other example starters, as well. Do you want me to remove this code from this particular starter?

Thanks for the additional input. Some of the comments, particularly those pertaining to the commented code, have not yet been resolved.

Hey @bgodlin, thanks for the review. I tried to reference all the source files that I've used. The commented out functions exist in other example starters, as well. Do you want me to remove this code from this particular starter?

Thanks for the additional input. Some of the comments, particularly those pertaining to the commented code, have not yet been resolved.

I see. No need to worry about this for now, then.

Thanks for your time @bgodlin!


add scripts to output k-mers and k-mer counts

An initial implementation of a simple script to output k-mer counts.
Per request by @drtamermansour.

Sweet. This is a great foundation for implementing #724

[ ] write a recipe for both scripts
[ ] create an issue to move the final for loop(s) into a C++ function as with abundance-dist.py

@mr-c @camillescott @luizirber please review at your leisure.

Otherwise, LGTM

Otherwise, LGTM

Points in https://github.com/dib-lab/khmer/pull/983#issuecomment-100564626 punted to issues; ready for review and merge @mr-c @luizirber @camillescott

LGTM, done


Drop down animation for Sticky headers

Hi thanks for your job!
I tried to use 5.0.0-b7 for sticky headers. But it looks like it doesn't  work correctly yet(list jums when it is scrolled).
Than I tried 5.0.0-SNAPSHOT. It works correctly, but dropdown animation appears when call updateDataSet method is called.
How can I disable this animation?

Hi @Bylynka, thank you!
But what do you mean with dropdown animation?

Please look at the video https://www.youtube.com/watch?v=da6dcHgjFrA
on 'PITCHES' tab I extend FlexibleAdapter on 'TOPICS' tab I extend RecyclerView.Adapter.
On 'PITCHES' tab you can see this animation( when updateDataSet is called).

What you see, is the insert of the headers that is done with a notification of notiyItemInserted().
showAllHeaders is also called in more parts of the adapter, to restore the initial set.
Actually you can't disable. I don't know if I can remove it, I will need to redesign some methods. But that effect is not that bad I think.

Thank you for answer! I agree with you that effect is not bad. But for current project I need list without any effects..

I will see what I can do, but I can't assure when it can be done.

Thank you. It will be great.

OK, yesterday I wanted to check something on this issue. For the moment the new code is valid only for updateDataSet(). Initialization works as before with the notifications.
In experiments branch I added the new code to skip all notifyItemXXX() only when animate is set false. Just before calling notifyDataSetChanged(), I complete the adapter set with expandables expanded and headers to be shown. Which means no effects at all when new items are fetched or items are removed from the list.
This can be obtained by calling updateDataSet(items) or updateDataSet(items, false). This was already in place but subItems and headers were applied after the notifyDataSetChanged(). Now they are added before.
In the demoApp I added a checkMenuBox to see the difference when animate = true or false:

When true it synchronizes all new and existent items with notifyItemXXX().
When false it doesn't make any animation, because notifyDataSetChanged() is called.

Great! I tried it and it exact I need. Will you add these updates to release?