Memory Leak

Hi,
Thanks a lot for your project, it works great but I'm facing with a memory leak.
I've compiled it on arm (Linux) an I push 2 stream over RTMP in FLV.
If I run it from shell with command:
sudo /home/ods/ods/mediaserver/MediaServer
it works like a charm
But if I run it from systemd something terrible happens and it starts to eat ram until it doesn't crash...
Systemd service file:

[Unit]
Description=MediaServer
After=network.target
[Service]
Restart=always
ExecStart=/home/ods/ods/mediaserver/MediaServer
[Install]
WantedBy=multi-user.target

Log rows during memory leak:

2020-05-29 10:28:15.455 W MediaServer[32057] EventPoller.cpp:156 modifyEvent | take time:8ms
2020-05-29 10:28:22.459 W MediaServer[32057] EventPoller.cpp:212 onPipeEvent | take time:6ms
2020-05-29 10:28:30.057 W MediaServer[32057] EventPoller.cpp:156 modifyEvent | take time:8ms
2020-05-29 10:28:40.714 W MediaServer[32057] EventPoller.cpp:156 modifyEvent | take time:6ms
2020-05-29 10:28:45.957 W MediaServer[32057] EventPoller.cpp:212 onPipeEvent | take time:8ms
2020-05-29 10:28:53.559 W MediaServer[32057] EventPoller.cpp:212 onPipeEvent | take time:9ms
2020-05-29 10:28:58.353 W MediaServer[32057] EventPoller.cpp:156 modifyEvent | take time:7ms
2020-05-29 10:29:00.261 W MediaServer[32057] EventPoller.cpp:212 onPipeEvent | take time:10ms
2020-05-29 10:29:04.225 W MediaServer[32057] EventPoller.cpp:212 onPipeEvent | take time:6ms
2020-05-29 10:29:09.955 W MediaServer[32057] EventPoller.cpp:212 onPipeEvent | take time:7ms
2020-05-29 10:29:11.214 W MediaServer[32057] EventPoller.cpp:156 modifyEvent | take time:13ms
2020-05-29 10:29:44.785 W MediaServer[32057] EventPoller.cpp:212 onPipeEvent | take time:6ms
2020-05-29 10:29:49.337 W MediaServer[32057] EventPoller.cpp:156 modifyEvent | take time:10ms
2020-05-29 10:29:56.226 W MediaServer[32057] EventPoller.cpp:156 modifyEvent | take time:14ms
2020-05-29 10:30:16.758 W MediaServer[32057] EventPoller.cpp:212 onPipeEvent | take time:13ms
2020-05-29 10:30:58.023 W MediaServer[32057] EventPoller.cpp:212 onPipeEvent | take time:10ms
2020-05-29 10:31:08.093 W MediaServer[32057] EventPoller.cpp:212 onPipeEvent | take time:8ms
2020-05-29 10:31:09.669 W MediaServer[32057] EventPoller.cpp:212 onPipeEvent | take time:6ms
2020-05-29 10:31:23.852 W MediaServer[32057] EventPoller.cpp:212 onPipeEvent | take time:8ms
2020-05-29 10:31:31.439 W MediaServer[32057] EventPoller.cpp:212 onPipeEvent | take time:11ms
2020-05-29 10:31:33.746 W MediaServer[32057] EventPoller.cpp:212 onPipeEvent | take time:9ms
2020-05-29 10:31:36.247 W MediaServer[32057] EventPoller.cpp:212 onPipeEvent | take time:7ms
2020-05-29 10:32:02.779 W MediaServer[32057] HttpSession.cpp:102 onError |<PHONE_NUMBER>88(<IP_ADDRESS>:2292) FLV播放器(defaultVhost/live/camera1)断开:end of file,耗时(s):255
2020-05-29 10:32:02.780 T MediaServer[32057] HttpSession.cpp:38 ~HttpSession |<PHONE_NUMBER>88(<IP_ADDRESS>:2292)
2020-05-29 10:32:22.780 W MediaServer[32057] MediaSource.cpp:442 operator() | 无人观看事件:rtmp/defaultVhost/live/camera1
2020-05-29 10:32:31.432 W MediaServer[32057] EventPoller.cpp:212 onPipeEvent | take time:6ms
2020-05-29 10:32:35.945 W MediaServer[32057] EventPoller.cpp:212 onPipeEvent | take time:6ms
2020-05-29 10:32:38.429 W MediaServer[32057] HttpSession.cpp:102 onError |<PHONE_NUMBER>12(<IP_ADDRESS>:2293) FLV播放器(defaultVhost/live/camera2)断开:end of file,耗时(s):285
2020-05-29 10:32:38.429 T MediaServer[32057] HttpSession.cpp:38 ~HttpSession |<PHONE_NUMBER>12(<IP_ADDRESS>:2293)
2020-05-29 10:32:45.822 W MediaServer[32057] EventPoller.cpp:156 modifyEvent | take time:9ms
2020-05-29 10:32:58.430 W MediaServer[32057] MediaSource.cpp:442 operator() | 无人观看事件:rtmp/defaultVhost/live/camera2
2020-05-29 10:33:01.696 W MediaServer[32057] EventPoller.cpp:156 modifyEvent | take time:6ms
2020-05-29 10:33:21.381 W MediaServer[32057] EventPoller.cpp:156 modifyEvent | take time:7ms
2020-05-29 10:33:26.259 W MediaServer[32057] EventPoller.cpp:156 modifyEvent | take time:18ms

Have you an idea how to solve it? I'm going crazy... :(

Which version of code do you use?  What is git hash？
You can use Valgrind to test and analyze the memory leak point of ZLMediaKit.

Which version of code do you use? What is git hash？
You can use Valgrind to test and analyze the memory leak point of ZLMediaKit.

cloned repository just now
Valgrind? I never used it, and unfortunately I'm not able to program in c++

How do you add cameras

I push streams using OpenCV and Python.

By RTMP ?

Yes in RTMP packed in FLV using Gstreamer.

codec is H264 + AAC  ?

You can run Valgrind like this：
$ valgrind ./MediaServer  --leak-check=full

When you exit MediaServer, valgrind will printf link this:
2020-05-29 16:56:37.543 I MediaServer[4732] main.cpp:358 start_main | 程序退出中,请等待...
2020-05-29 16:56:38.546 I MediaServer[4732] main.cpp:360 start_main | 程序退出完毕!
2020-05-29 16:56:38.578 I MediaServer[4732] EventPoller.cpp:94 ~EventPoller | 0x631a9f0
2020-05-29 16:56:38.585 I MediaServer[4732] EventPoller.cpp:94 ~EventPoller | 0x631bb80
2020-05-29 16:56:38.586 I MediaServer[4732] EventPoller.cpp:94 ~EventPoller | 0x631c790
2020-05-29 16:56:38.589 I MediaServer[4732] EventPoller.cpp:94 ~EventPoller | 0x631d3c0
2020-05-29 16:56:38.591 I MediaServer[4732] EventPoller.cpp:94 ~EventPoller | 0x631e7a0
2020-05-29 16:56:38.592 I MediaServer[4732] EventPoller.cpp:94 ~EventPoller | 0x631f410
2020-05-29 16:56:38.594 I MediaServer[4732] EventPoller.cpp:94 ~EventPoller | 0x631ffc0
2020-05-29 16:56:38.596 I MediaServer[4732] EventPoller.cpp:94 ~EventPoller | 0x6320b70
2020-05-29 16:56:38.816 I MediaServer[4732] logger.cpp:57 ~Logger |
==4732==
==4732== HEAP SUMMARY:
==4732==     in use at exit: 1,472 bytes in 22 blocks
==4732==   total heap usage: 483,040 allocs, 483,018 frees, 242,941,502 bytes allocated
==4732==
==4732== 608 bytes in 1 blocks are possibly lost in loss record 22 of 22
==4732==    at 0x4A07F5C: calloc (vg_replace_malloc.c:752)
==4732==    by 0x355CA11D02: _dl_allocate_tls (in /lib64/ld-2.12.so)
==4732==    by 0x355D2072CC: pthread_create@@GLIBC_2.2.5 (in /lib64/libpthread-2.12.so)
==4732==    by 0xD8B872: std::thread::_M_start_thread(std::shared_ptr<std::thread::_Impl_base>, void (*)()) (in /media/psf/git/ZLMediaKit/release/linux/Debug/MediaServer)
==4732==    by 0xD65322: std::thread::thread<toolkit::initMillisecondThread()::{lambda()#1}>(toolkit::initMillisecondThread()::{lambda()#1}&&) (in /media/psf/git/ZLMediaKit/release/linux/Debug/MediaServer)
==4732==    by 0xD64EB5: toolkit::initMillisecondThread() (in /media/psf/git/ZLMediaKit/release/linux/Debug/MediaServer)
==4732==    by 0xD650C9: toolkit::getCurrentMicrosecond() (in /media/psf/git/ZLMediaKit/release/linux/Debug/MediaServer)
==4732==    by 0xD37EE9: toolkit::ThreadLoadCounter::ThreadLoadCounter(unsigned long, unsigned long) (in /media/psf/git/ZLMediaKit/release/linux/Debug/MediaServer)
==4732==    by 0xD38170: toolkit::TaskExecutor::TaskExecutor(unsigned long, unsigned long) (in /media/psf/git/ZLMediaKit/release/linux/Debug/MediaServer)
==4732==    by 0xD34647: toolkit::EventPoller::EventPoller(toolkit::ThreadPool::Priority) (in /media/psf/git/ZLMediaKit/release/linux/Debug/MediaServer)
==4732==    by 0xD366AC: toolkit::EventPollerPool::EventPollerPool()::{lambda()#1}::operator()() const (in /media/psf/git/ZLMediaKit/release/linux/Debug/MediaServer)
==4732==    by 0xD36C2D: void toolkit::TaskExecutorGetterImp::createThreads<toolkit::EventPollerPool::EventPollerPool()::{lambda()#1}>(toolkit::EventPollerPool::EventPollerPool()::{lambda()#1}&&, int) (in /media/psf/git/ZLMediaKit/release/linux/Debug/MediaServer)
==4732==
==4732== LEAK SUMMARY:
==4732==    definitely lost: 0 bytes in 0 blocks
==4732==    indirectly lost: 0 bytes in 0 blocks
==4732==      possibly lost: 608 bytes in 1 blocks
==4732==    still reachable: 864 bytes in 21 blocks
==4732==         suppressed: 0 bytes in 0 blocks
==4732== Reachable blocks (those to which a pointer was found) are not shown.
==4732== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==4732==
==4732== For counts of detected and suppressed errors, rerun with: -v
==4732== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 4 from 4)

In fact, I didn't find any real memory leaks in zlmediakit.
I just used ffmpeg to push RTMP to zlmediakit, and then used ffplay to play RTMP and HTTP-flv.

when I run in shell it uses only 0.5% of 4GB memory.
when I run it as service it starts immediately to eat all ram.
I've installed valgrind but it returns:

root@localhost:/home/ods/ods/mediaserver# valgrind ./MediaServer  --leak-check=full
==2856== Memcheck, a memory error detector
==2856== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==2856== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==2856== Command: ./MediaServer --leak-check=full
==2856==
未识别的选项,输入"-h"获取帮助.
==2856==
==2856== HEAP SUMMARY:
==2856==     in use at exit: 0 bytes in 0 blocks
==2856==   total heap usage: 314 allocs, 314 frees, 105,696 bytes allocated
==2856==
==2856== All heap blocks were freed -- no leaks are possible
==2856==
==2856== For counts of detected and suppressed errors, rerun with: -v
==2856== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

sorry , the cmd is:
$ valgrind  --leak-check=full ./MediaServer

When you have finished testing, you can press Ctrl + C in the shell, or enter exit, and then watch Valgrind's log.

Here it is:

==2902==
==2902== HEAP SUMMARY:
==2902==     in use at exit: 304 bytes in 2 blocks
==2902==   total heap usage: 688,031 allocs, 688,029 frees, 687,486,276 bytes allocated
==2902==
==2902== 288 bytes in 1 blocks are possibly lost in loss record 2 of 2
==2902==    at 0x4846B0C: calloc (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)
==2902==
==2902== LEAK SUMMARY:
==2902==    definitely lost: 0 bytes in 0 blocks
==2902==    indirectly lost: 0 bytes in 0 blocks
==2902==      possibly lost: 288 bytes in 1 blocks
==2902==    still reachable: 16 bytes in 1 blocks
==2902==         suppressed: 0 bytes in 0 blocks
==2902== Reachable blocks (those to which a pointer was found) are not shown.
==2902== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==2902==
==2902== For counts of detected and suppressed errors, rerun with: -v
==2902== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

According to Valgrind's logs, there is no memory leak

Have you changed the code of zlmediakit?
Do you have a high CPU load?

When your computer frantically prints this log:
2020-05-29 10:29:00.261 W MediaServer[32057] EventPoller.cpp:212 onPipeEvent | take time:10ms

It means you have a high CPU load

when CPU load is high, which may lead to memory backlog and exhaustion。

Nothing touched, I only use webhook in order to obtain events.
Not a big CPU usage, around 17-20%

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
4451 root      20   0 1719700 132424  51220 S  68,2  3,3   0:22.81 /usr/bin/python3 /home/ods/ods/detector/viewer.py camera2
4452 root      20   0 1719404 154324  50520 S  63,9  3,8   0:22.24 /usr/bin/python3 /home/ods/ods/detector/viewer.py camera1
4467 root      20   0 10,629g<PHONE_NUMBER>76 S  38,4 19,0   0:16.24 /usr/bin/python3 /home/ods/ods/detector/detector.py camera2
4466 root      20   0 10,628g<PHONE_NUMBER>68 S  28,9 18,8   0:13.78 /usr/bin/python3 /home/ods/ods/detector/detector.py camera1
4394 root      20   0  759688  37284   8996 S  19,0  0,9   0:06.94 ./MediaServer

Please use this api:  http://<IP_ADDRESS>/index/api/getThreadsLoad
to Monitor the load on zlmediakit and tell me the results.

{
"code" : 0,
"data" : [
{
"delay" : 0,
"load" : 0
},
{
"delay" : 1,
"load" : 20
},
{
"delay" : 0,
"load" : 0
},
{
"delay" : 0,
"load" : 0
}
]
}
{
"code" : 0,
"data" : [
{
"delay" : 0,
"load" : 0
},
{
"delay" : 0,
"load" : 10
},
{
"delay" : 0,
"load" : 0
},
{
"delay" : 0,
"load" : 0
}
]
}
{
"code" : 0,
"data" : [
{
"delay" : 0,
"load" : 0
},
{
"delay" : 0,
"load" : 16
},
{
"delay" : 0,
"load" : 0
},
{
"delay" : 0,
"load" : 0
}
]
}

The load is very low, which should not be the reason

Valgind said that there is no memory leak and the load is very low, so there should be no problem.

Can be something related to the platform?

@PaserSRL  Adn can you answer this question?

@PaserSRL Adn can you answer this question?

around 40-45 minutes in order to full 1.3GB of memory

This is a server which running zlmediakit on arm cpu: http://zlmediakit.com:8880/#/
And  it is stable.

root@NanoPi-R1S-H5:~# cat /proc/cpuinfo
processor	: 0
BogoMIPS	: 48.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

my own:

ods@localhost:~/ods/mediaserver_old$ cat /proc/cpuinfo
processor       : 0
model name      : ARMv8 Processor rev 1 (v8l)
BogoMIPS        : 38.40
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x1
CPU part        : 0xd07
CPU revision    : 1
processor       : 1
model name      : ARMv8 Processor rev 1 (v8l)
BogoMIPS        : 38.40
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x1
CPU part        : 0xd07
CPU revision    : 1
processor       : 2
model name      : ARMv8 Processor rev 1 (v8l)
BogoMIPS        : 38.40
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x1
CPU part        : 0xd07
CPU revision    : 1
processor       : 3
model name      : ARMv8 Processor rev 1 (v8l)
BogoMIPS        : 38.40
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x1
CPU part        : 0xd07
CPU revision    : 1

root@NanoPi-R1S-H5:~/git/ZLMediaKit/release/linux/Debug# file MediaServer
MediaServer: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=99af883b6659a590722a82504d838cd4caebfe3d, not stripped

My system is Ubuntu 16. I suspect that the kernel is older and buggy.

My board has ubuntu 18.04

would you like to connect in ssh to check yourself?

Please modify code like this:

Please modify code like this:

You try this first, SSH when it doesn't work

modified
recompiled

but problem is still there... :(

Uhm, wait a moment... let me test it better.

No, problem is still there...

Please check the video:
Live Video:
https://youtu.be/3WpJhtWrlCs
Log
ZLMediaKit Memory Leak Log.txt
Config:
config.txt
A little explanation:
I run a node JS app written by me that prepare the environment, it:

stop mediaserver (sudo service mediaserver stop)
start mediaserver (sudo service mediaserver start)
start 4 python processes, 2 viewer.py push stream on RTMP server.

I can SSH your board to debug it ?

Sorry for my late reply but it was holiday in Italy.
I found where is the bug, it seems related to HLS.
In fact if I disable HLS conversion it works properly.
Friday I've disabled HLS and it worked properly for 4 days.
Just for information: I've created a 128MB ramdisk with mounting point "/mnt/ramdisk" and I've configured hls.filePath to '/mnt/ramdisk/'.

You may do this repeated:
           System::execute("echo 1 > /proc/sys/vm/drop_caches");
            System::execute("echo 2 > /proc/sys/vm/drop_caches");
            System::execute("echo 3 > /proc/sys/vm/drop_caches");

My application executes:
"sync && sudo sh -c "echo 3 > /proc/sys/vm/drop_caches""
every 5 minutes

So zlmediakit has no memory leak?

So, zlmediakit has no memory leak?

Yes it has, if you enable HLS function it will generate a memory leak.
If you disable it problem disappear.

You may need to do this repeated:
System::execute("echo 1 > /proc/sys/vm/drop_caches");
System::execute("echo 2 > /proc/sys/vm/drop_caches");
System::execute("echo 3 > /proc/sys/vm/drop_caches");

I used to use ramdisk to make HLS root folder, and I met the same problem with you.
Then, i repeated execute the code above, and memory will no longer growing.

Can you give this project a star? thank you！

Can you give this project a star? thank you！

Absolutely yes! Your application is wonderful!
I've implemented your suggestion, now I'm testing it.

[428267.541588] Out of memory: Kill process 761 (MediaServer) score 308 or sacrifice child
[428267.557103] Killed process 761 (MediaServer) total-vm:2660984kB, anon-rss:971552kB, file-rss:0kB, shmem-rss:0kB
[428268.642919] oom_reaper: reaped process 761 (MediaServer), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Unfortunately it didn't worked...


Yield fails a couple of JS Benchmarks

[ ] crypto-aes
[ ] string-unpack-code

@rachitnigam , can you verify these fail for you too and take a look at what's going on with yield? It may be a common mistake between yield and CPS, since there's overlap in the tests that fail between them

crypto-aes uses builtin HOFs for String.replace

These both use replace with HOFs. Closing now that #21 addresses the real issue.


Add signing to deb and rpm builds

This PR ensures that the deb and rpm linux artifacts are being signed in the build pipeline.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 41.574%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
30787

Relevant Lines:
69355

💛  - Coveralls

Saw that the steps in the linux pipeline to sign pass now after the latest changes! Did you also get a chance to check that the files from that build are signed as expected?

Converting to draft, as we will be updating signing keys to new one once they're active in early Dec.


Using Bootstrap 3, my take

I need to use Bootstrap 3 in my test app which is based on the minimal branch. So, these are my changes:

npm install bootstrap-loader
npm install jquery --save-dev
npm install bootstrap-sass css-to-string-loader<EMAIL_ADDRESS>file-loader sass-loader style-loader url-loader --save-dev

constant.ts:
const ProvidePlugin = require('webpack/lib/ProvidePlugin');

exports.MY_CLIENT_PLUGINS = [
  // use this to import your own webpack config Client plugins.
   new ProvidePlugin({
            jQuery: 'jquery',
            $: 'jquery',
            jquery: 'jquery',
            "Tether": 'tether',
            "window.Tether": "tether"
        })

]

main.browser.ts:
import 'jquery';
import 'bootstrap-loader';

app.component.html (just to see if Bootstrap works):

Angular Starter App

			{{view.name}}

				<button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1"
					aria-expanded="false">
        Toggle navigation

      </button>
				Brand

Link (current)

Link

						Dropdown

Action

Another action

Something else here

Separated link

One more separated link

Link

						Dropdown

Action

Another action

Something else here

Separated link

		<router-outlet (activate)="activateEvent($event)" (deactivate)="deactivateEvent($event)"></router-outlet>

rename app.component.css to app.component.scss and add this import:
@import "~bootstrap-sass/assets/stylesheets/bootstrap";

app.component.ts - change accordingly:
  styleUrls: ['./app.component.scss'],

webpack.config.ts - add loaders:
      {
        test: /\.(sass|scss)$/,
        loaders: ['css-to-string-loader', 'css-loader?sourceMap', 'resolve-url', 'sass-loader?sourceMap']
      },
      { test: /\.woff(2)?(\?v=[0-9]\.[0-9]\.[0-9])?$/, loader: 'url?limit=10000&minetype=application/font-woff' },
      { test: /\.(ttf|eot|svg)(\?v=[0-9]\.[0-9]\.[0-9])?$/, loader: 'file' },

Now, since I'm not even close to webpack (and AOT, DLLs, etc) expert, my question - is this correct approach? I see that it works in dev and prod, haven't tried AOT yet.
Can we have a separate branch for Bootstrap 3? I may try to use ng2-bootstrap later..

@alvipeo well directly using scss instead of scss will make AOT compile fail, which on the Other branches you will see I'm css for the styles URL but working with scss that is being converted with node-sass...(this issue will change once we can use the ngtools/webpack plugin). Beyond that, I suppose using JQuery at all may make it problematic to use Universal...not quite sure about AOT.
I'm not going to make a Bootstrap 3/Jquery branch as I think it's generally recommended to avoid using JQuery as you're probably going to run into a ton of issue when it comes to Universal, Web Workers and possibly AOT. I was working on a boostrap 4 branch that uses ng2-bootstrap, but they currently have an issue with AOT (which should be solved eventually).

Well, some people do not need Universal, that's why I used minimal branch.
I'd go for a Bootstrap 4, but it seems like its alpha state is never ending story, so I'll skip. ng2-bootstrap looks good, I'm going to try it.
Didn't get what you said about scss... at all ;)

@alvipeo well of course, some people won't need to use Universal or Web Workers and maybe won't use AOT, but as using JQuery can be very problematic and limiting, I rather not create a branch based around using it.
As far as scss, the ngc compiler does not support or understand sass...so using it in your styles url will cause AOT compilation to fail. In the other branches, you'll see that I mitigate this by compiling the scss using node-sass and referencing the css that is output in the stylesUrl array. There is going to be a webpack plugin for AOT that will get around this, but it's not available for public use yet.

ok, thanks. I'd pass Bootstrap, but what are options? Bootstrap 4 is not even close to ready, Material is not going to be even at beta by the end of the year... so how would one make a web ui really? This I think is the main reason people use React.

@alvipeo If you're just using Bootstrap 4 for it's css, I wouldn't worry about it as the api for that is pretty stable. Combine that with ng2-bootstrap which should become AOT compatible soon, and you're golden.

ng-bootstrap*
and it's already aot ready

@DzmitryShylovich yeah, ng-bootstrap is the correct repo 😄
so this issue should be closed now? https://github.com/ng-bootstrap/ng-bootstrap/issues/796

Hmm I thought it work in aot. My bad.

@DzmitryShylovich yeah, it still doesn't appear to work just yet, it's not exporting all of the .ngfactory files that it needs to
I just added the bootstrap branch I was working on locally... whenever they fix AOT on ng-bootstrap, I'll add the fix there

wow! Thanks you guys for Bootstrap 4 branch!


Fixed #33544 -- Added hint to default behavoir of cached template loader.

The TEMPLATES section of the Deployment checklist lacked a hint to the default behavior of the cached template loader.
It didn't point out that the cached template loader is enabled only when DEBUG = False.
The newly added sentence clarifies that behavoir and makes the section be in consistence with others (for example ALLOWED_HOSTS)

@carltongibson, is this okay for now?


Add app tag to all cloud resources

What
Add app tag to all cloud resources that matches the project name via terraform

Completed in #38


add some description to logout

Coverage remained the same at 73.975% when pulling ccec3ef5d994e42e8c42fbc3ae224f80c62d6a81 on rotembr-patch-2 into e2218cd7c18cdece33864ae3ee5fdbd945e4fdb5 on development.


dom api implementation

resolves https://github.com/paldepind/snabbdom/issues/47

The idea is anyone can implement this interface
module.exports = {
  createElement: createElement,
  createElementNS: createElementNS,
  createTextNode: createTextNode,
  appendChild: appendChild,
  removeChild: removeChild,
  insertBefore: insertBefore,
  parentNode: parentNode,
  nextSibling: nextSibling,
  tagName: tagName,
  setTextContent: setTextContent
};

and use snabbdom with any type of dom.

I like this. Similar to the changes I made to add  support (see #72), but more flexible. I think I could re-implement what's in #72 without having to change the core for people who don't care about iframes.

I'll be that guy who does a :+1:

This is nice. What are you planning to use this for @ptol?
I'm a bit hesitant to merge this as I'm not sure if and how people will use it?
@rayd That sounds like a really good idea. Iframe support is nice but also rarely needed.

:+1: from me.  Was just looking at the feasibility of rendering vnodes to something like blessed (ncurses), this would help I think.

@paldepind There are dozen of js libraries with their own dom (all game engines and canvas/webgl libraries) so if someone need a virtual dom for such library the snabbdom would be a possible solution.
I have been using it with pixi.js
var nodeTypes = {};
nodeTypes.text = () => new PIXI.Text("", {font: "14px Verdana", fill: "white"});
nodeTypes.group = () => new PIXI.Container();
nodeTypes.sprite = () => new PIXI.Sprite(px.Texture.EMPTY);
...

function createElement(tagName){
  var node = nodeTypes[tagName]();
  if(node){
    node.tagName = tagName;
  }
  return node;
}

function insertBefore(parentNode, newNode, referenceNode){
    if(referenceNode){
      var i = parentNode.getChildIndex(referenceNode);
      parentNode.addChildAt(newNode,i);
    }else{
      parentNode.addChild(newNode);
    }
}

function removeChild(node, child){
  node.removeChild(child);
}

function appendChild(node, child){
  node.addChild(child);
}

function parentNode(node){
  return node.parent;
}

function nextSibling(node){
  var parent = node.parent;
  var i = parent.getChildIndex(node);
  var sublingIndex = i + 1;
  if(parent.children.length <= sublingIndex) return null;
  return parent.getChildAt(sublingIndex);
}

function tagName(node){
  return node.tagName;
}

module.exports = {
  createElement: createElement,
  appendChild: appendChild,
  removeChild: removeChild,
  insertBefore: insertBefore,
  parentNode: parentNode,
  nextSibling: nextSibling,
  tagName: tagName
};

@paldepind Or do you have something different in mind for how this https://github.com/paldepind/snabbdom/issues/47 could be solved?

@ptol Nope. This definitely seems to be the way to solve #47. The above is exactly what I wanted to see. A proof of concept that this abstracted interface can actually be usefully implemented by anything besides the DOM. Using it to control the pixi.js scene graph is certainly useful.

@ptol Thank you a lot for this PR! As you can see I've merged it.
Unless I'm mistaken then the code in this PR would force all the patch functions returned by init to use the same DOM API. I've made a commit that should keep the different APIs separate. So that one, for instance, can use Snabbdom to patch both the DOM and pixi.js on the same page with different patch instances. Will you try it ou @ptol? After that I'll do a release.
Thanks again :smile:

Unless I'm mistaken then the code in this PR would force all the patch functions returned by init to use the same DOM API.
I don't think so,  domApi is a local variable

var domApi = require('./htmldomapi.js');

so
var patch1 = snabbdom.init([], require('./pixiapi')); //pixi
var patch2 = snabbdom.init([]); //html

shouldn't affect each other.

I've made a commit that should keep the different APIs separate

Do you mean this commit https://github.com/paldepind/snabbdom/commit/fb80c1fd80235e93803d51a7ab274d659fd1a317 ?
I don't see any related changes.

No strong feelings about it, but what about a new patch constructor instead of overloading init ?
I don't see any related changes.

I've changed it so that api is local to the function returned by patch and move a few function that use api so that they are inside the created closure.
@ericgj

No strong feelings about it, but what about a new patch constructor instead of overloading init ?  Also to me it feels like the api should be the first parameter, and the modules the second.

A new initWithApi function wouldn't bother me—but I'm indifferent. I don't mind the overloading here but I can see the appeal in avoiding it. Why do you feel the API argument should be first?

Oh, I see. api should definitely be in function scope. I used last version with different dom on one page and everything work fine.

Released and published to npm.
@ptol If you feel like it I think publishing your pixi.js implementation of the DOM interface would be really great! We can create a folder in the repo for API implementations or you can create a repository of your own and I will link to it from the Snabbdom README.

@paldepind Yes, I have plans to create a repository but I need to fix some problems before.
In canvas/webgl libraries there is a method destroy. This method need to be called when you don't need the object anymore.
var vnode1 = h("group", {}, [h("text", {props: {text: "foo"}} )]);
var vnode1 = h("group", {}, []);

For example, this text object should be destroyed. What would be the best place to call this method?
Can I do this?
function removeChild(node, child){
  node.removeChild(child);
  child.destory();
}

Or even it's removed this child can still be used somewhere?

Here the repository https://github.com/ptol/virtual-pixi
It seems there isn't any problem with
function removeChild(node, child){
  node.removeChild(child);
  child.destory();
}

Hi,
This is a great valuable addition which require to be shown in the doc.
I use snabbdom for a while, however this is our google friend that driven me on that issue when I was looking for virtualdom tool for NativeScript.
Cheers,


Implement a NDWI filter to reduce aerosol-derived noise

In the current version of the DE Africa Coastlines code, ARD processing issues believed to be due to aerosols cause significant noise in the output shorelines:

To reduce this noise noise over open water, we can apply an additional mask based on NDWI. This works because NIR is less affected by the aerosol issues than SWIR, and NDWI tends to be less aggressive at mapping water than MNDWI, which ensures that masking by NDWI will not remove useful along the actual coastline. NDWI water pixels are first identified then shrunk using erosion to ensure only open water is modified:

This PR also makes some minor fixes to ensure notebooks run correctly on the Sandbox.

@alexgleith If 50 tiles isn't too many, this would be a good set of tile IDs (not indices) to run as a test:
['39', '41', '45', '46', '51', '86', '87', '129', '137', '138', '145', '190', '200', '302', '327', '329', '330', '399', '411', '474', '475', '479', '486', '529', '608', '624', '630', '652', '653', '1077', '1083', '1105', '1107', '1153', '1198', '1199', '1209', '1212', '1257', '1286', '1309', '1319', '1332', '1339', '1340', '1348', '1350', '1367', '1469', '1499', '1705']

I've run this, @robbibt.
The following IDs failed: 327, 329, 330`
All of them failed with:

rasterio.errors.RasterioIOError: '/vsis3/deafrica-landsat/collection02/level-2/standard/tm/1998/202/035/LT05_L2SP_202035_19980104_20200909_02_T1/LT05_L2SP_202035_
19980104_20200909_02_T1_SR_B4.TIF' not recognized as a supported file format.

This will only be an issue in DEV, and I've archived the dataset 09dd59e0-54c2-5bb7-b1e4-f8357cd0c8f2 to resolve it.
And, data is here: https://deafrica-data-dev-af.s3.af-south-1.amazonaws.com/coastlines/v0.1.1/coastlines_v0.1.1.gpkg


rock_paper_scissor (game)

Description
Added a script (.py) file for the Rock Paper Scissor game.
It takes input from the user and generates a random input for CPU, and compares the result and generates the output accordingly.

user input (player 1).
comparing the input with random self-generated input by CPU(player 2).
Displaying the result.
Asking the user whether he wishes to continue the game or not.
(For each iteration, it asks the user whether the user wishes to continue or not )
code is developed using python 3.

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes # (issue)
Type of change
Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[ ] Test A
[ ] Test B

Test Configuration:

Firmware version:
Hardware:
Toolchain:
SDK:

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] Any dependent changes have been merged and published in downstream modules

Edit the description to fit your PR.

updated

Include resources in readme.md file as they are added for existing projects.
No plagiarism. If you have copied code, mention the resource in README.
Working code. We will check all codes functionality so they should be working and fulfil the purpose.
Separate folder by correct name and inside it all the files. Make a separate folder like URL_SHORTENER and put all the python and related files in it even if there's a single file.
Also include screenshot in the folder. Optional but if you want you can include screenshots of your working code in the same folder as above.

P.S. You dont need to close your existing PRs. Just push the changes in your forked repo and they will automatically get updated in my repo.
FOLLOW THESE STEPS FOR FUTURE PRs ALSO.

@pranesh-badarinath  did u make the changes? The timeline is 1 week per project. It's way past the timeline.

By today evening I ll updade the changes.
On Fri, 7 Aug, 2020, 12:00 AM Garima Singh<EMAIL_ADDRESS>wrote:

@pranesh-badarinath https://github.com/pranesh-badarinath did u make
the changes? The timeline is 1 week per project. It's way past the timeline.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/garimasingh128/awesome-python-projects/pull/22#issuecomment-670100971,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AONNLKAGBW4JH63NFPF2YM3R7LZF3ANCNFSM4PPTHJGA
.

Perfect! Good job

thank you

@pranesh-badarinath I think you forgot to add resources in README. Create a separate PR to just add some resources!


Feature: Mirror "Reveal in Tree View" feature

I'm using Atom 1.7.3 on Linux and when I right click inside of a code pane it gives me the option to reveal the current file's location in the tree view.  It would be groovy if we could do the same thing using remote-ftp.  The menu option could reside in either the right click context menu or the file name tab.  Clicking it would un-collapse the parent folders and select the file in the remote-ftp pane.

+1 for this


imresize is deprecated, imresize is deprecated in SciPy

'imresize()' is deprecated! imresize is deprecated in SciPy 1.0.0, and it was removed in new version (1.2.0)
need to change:

replacing imresize with skimage.transform.resize
verifying training results

current error:
File "/root/src/rl_coach/filters/observation/observation_rescale_to_size_filter.py", line 83, in filter
observation = scipy.misc.imresize(observation,

AttributeError: module 'scipy' has no attribute 'misc'

PR #321 fixes this issue.

Fixed with #321


Page rank contributors

Run page rank algorithm on developer social network.
Network considers relationships from mutual commit authorship, PR message threads, Issue message threads, and PR reviews.
Goal is to identify, in order, the highest to lowest ranked contributors.
Weights of interactions ( co-committership, PR reviewing, etc.) should, for the time being, be naive. (all interactions should have a weight of 1.0)

Notebook: https://colab.research.google.com/drive/193jHnC_WXCO9UymOTNitbLhBW9LBvows#scrollTo=y7BOZWtPTKja
Top 10 conributors based on pagerank (with naive weights):

01012f1b-7f00-0000-0000-000000000000: Score 0.08243858925165543
01000045-2400-0000-0000-000000000000: Score 0.07939057787454838
01000c4d-d800-0000-0000-000000000000: Score 0.0701451505214677
01000cc2-4b00-0000-0000-000000000000: Score 0.037363315745449786
01000067-2300-0000-0000-000000000000: Score 0.028521744677016084
01000e5a-0d00-0000-0000-000000000000: Score 0.021280848463192666
01000099-ac00-0000-0000-000000000000: Score 0.016306323695686374
01001c87-8900-0000-0000-000000000000: Score 0.015489393733754565
01000331-5a00-0000-0000-000000000000: Score 0.014844764808603472
010009ab-a500-0000-0000-000000000000: Score 0.012833322285940483

Will add this to adjacency matrix

Closed as in "done" column


InvalidBEncodingException: Unknown indicator '239'

Hi,
I have the following error:
   com.turn.ttorrent.bcodec.InvalidBEncodingException: Unknown indicator '239'

when I try to open these files:
https://yts.to/torrent/download/A779AED378D58D97172E176BA7AB5250298BB0AA.torrent
https://yts.to/torrent/download/60B101018A32FBDDC264C1A2EB7B7E9A99DBFB6A.torrent
The code is very simple:
// Torrent download
if (! Utils.urlContentsToFile(url, torrentFile)) {
    download.incrementErrorCount();
    return;
}

Client client = null;
try {
    client = new Client(InetAddress.getLocalHost(), SharedTorrent.fromFile(torrentFile, downloadDir));
} catch (IOException e) {
    e.printStackTrace();
    download.incrementErrorCount();
    continue;
 }

client.download();
client.waitForCompletion();

I don't know if these torrents are using a bad format (I don't think so because Transmission opens them without problem) or a newer format, or I am doing something wrong.
Thank you in advance!

I have been debugging the parsing process of the torrent file and the exception occurs after reading the last element "pieces". It reads the whole buffer but the next byte readed is '239'. I couldn't find where this byte is in the file because no '239' byte is near the end of the 'pieces' element.
I also tried to open it with the CLI tool and it works perfectly so I think I am doing something wrong. Any idea?
BTW, I am using the library in an Android application development. I don't know if it could be related

I found the error. As I thought it was my fault. I was storing the data on the torrent file from network using string readings. It converted data into UTF-8 so it corrupted the file.
Sorry for the inconveniences


Q: How to change resolution/qualitiy

How can I change the quality of youtube videos that are played to kodi? Currently it plays 640x360 on a 1080p screen (HDTV) no matter how I start it from firefox.

Today it works. :)
I don't know if it is because I changed some settings in the youtube addon in Kodi or if it is video related (both videos I tested support 1080p in YT)

@maulwuff Cast Kodi only sends the video ID to the YouTube add-on in Kodi. Video quality can be configured in the Settings of the YouTube add-on in Kodi.


Using ice4j behind ALB in a VPC at AWS (Kubernetes)

First of all, thank you so much for this awesome work! it has been really nice working with this tool.
We have a problem though. Our infrastructure runs behind an Application Load Balancer inside a VPC in a Kubernetes cluster. The VPC has an external IP.
The requests are something like this:
pod:<IP_ADDRESS>
ec2Instance: <IP_ADDRESS>
eni: <IP_ADDRESS> # Example IP
alb:
  - <IP_ADDRESS>
  - <IP_ADDRESS>

User -> alb (<IP_ADDRESS>) -> ec2Instance -> pod

The discovery from the pod to find the external IP works, as can be seen from the logs:
INFO: Using org.ice4j.ice.harvest.StunMappingCandidateHarvester, face=/<IP_ADDRESS>, mask=/<IP_ADDRESS>
Jun 03, 2020 6:52:18 AM org.ice4j.ice.harvest.MappingCandidateHarvesters initialize

However, as mentioned before, the IPs the user is hitting, are the IP's from the ALB. And those are different from the one the pod is using to go to internet.
We cannot "hard-code" the IPs of the ALB in the configuration because those are elastic. AWS will change them as the load in our system increases or decreases.
And the IP of the ENI does not allow access from the outside, is just from within the VPC out.
Is there any way to use the DNS name to do the discovery?
Thank you for your time and hard work!

Thanks for your quick answer @bgrozev !
The TTL for the DNS in the ALB is 60 seconds. This is to quickly react to changes in the traffic.
https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/how-elastic-load-balancing-works.html#request-routing

You could have a timer running rediscovery every minute

I'm not really a java programmer. I can see the Stun harvester is done here:
https://github.com/jitsi/ice4j/blob/master/src/main/java/org/ice4j/ice/harvest/StunMappingCandidateHarvester.java#L70
Could you point me in the right direction?

Just open it here and we'll take a look

@bgrozev , we're still working on this and researching our options. Question, can I add multiple IP addresses here:
NAT_HARVESTER_PUBLIC_ADDRESS

@bgrozev, I don't think we are going to need this. At least not for the time being. We found a different way to do it.
You can see it here.
We are still figuring how to make this more resilient to changes in the NLB. But maybe using (in our specific case) octo for Jitsi will help us to restart the video bridges without loosing the whole conference.
However, we will still be willing to write our own harvester in the future. But for the time being, this seems to be good enough four our use case.
I will not close the issue, but if you feel is better to, feel free to do it.
Thank you so much for your hard work!

@bgrozev , so after a few months more testing and working to do our release, we are in the staging phase. And it seems we might need to do this in the end.
Has it been any work on your side? We are looking into how to fix this (either by adding this code here or doing it externally)

@citosid there's been no work on this from our side.

Ok. Will keep looking into it and will let you know.
Thank you so much!!!

@citosid What about if we made it possible to specify a DNS name for NAT_HARVESTER_PUBLIC_ADDRESS (e.g. k8s-default-jvb-abcdef-abcdef.elb.us-east-1.amazonaws.com), and when starting ICE, the DNS name is resolved? This way you'd always have the current IP(s) for the load balancer.


Do we have a conclusion on which other joints the model relies on when predicting each joint of the human body?

hello，can u run this on windows？


Document that JAEGER_SAMPLER_MANAGER_HOST_PORT should be a URL

Right now the documentation says: The host name and port when using the remote controlled sampler. This is super vague and this led me to believe I should set it to: jaeger-agent.jaeger.svc:5778, when infact it should be: http://jaeger-agent.jaeger.svc:5778/sampling
Further, the error here is being just dropped on the floor: https://github.com/jaegertracing/jaeger-client-go/blob/master/sampler.go#L397-L399 making debugging super hard.

#327 for the error log part


Recognize short form of --help; don't ignore unknown arguments

After seeing #41 I realised my *) was naive.
I tried a different solution: include all arguments in the case statement, matching on prefix.
This is a bit more verbose, but I think clearer and less error-prone. (And the verbosity could be pushed into a function if need be.)

Also fixes #42

Thanks!


Build(deps-dev): Bump webpack-dev-server from 4.15.1 to 5.0.2 (#5563)

Pull request opened by the merge tool on behalf of #5563

Merged in 87561a22458efe485a7b505d36616a980ba48e5d


/run/cryptsetup is missing!

Describe the bug
Seemingly harmless warning being displayed.
WARNING: Locking directory /run/cryptsetup is missing!
I first noticed this on unstable while doing a clean install for a new machine. Tested again on
the 20.03 graphical installer live system just to make sure it wasn't an isolated case on the unstable.
I haven't tested this fully
To Reproduce
Steps to reproduce the behavior:

Run any of the installers
Run cryptsetup on a fresh partition
Complete the password prompt phase

Expected behavior
There should be no warning after the password prompt.
Screenshots

Notify maintainers
Metadata

system: "x86_64-linux"
host os: Linux 5.4.35, NixOS, 20.03.1577.74a80c5a9ab (Markhor)
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.3.4
channels(root): "nixos-20.03.1577.74a80c5a9ab"
nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

This occurred in the past and was resolved for a period of months through https://github.com/NixOS/nixpkgs/pull/60235. I have also seen this a few times the past month during some re installations.
Though it's pretty much harmless, since the directory is created after the use usage of the command (at least from when I created that PR).


In batch-mode consumer, for messages with collection payload, NegotiatingMessageConverterWrapper cannot divide headers for each element.

Is your feature request related to a problem? Please describe.
I used spring-cloud-stream 3.0.3.RELEASE and recently upgraded to spring-cloud-stream 3.0.8.RELEASE.
I am using a custom message converter that inherits AbstractMessageConverter and the stream function signature is Function<Flux<List<MyClass>>, Flux<Message<MyResultClass>>>.
I found my message being converted through the SimpleFunctionRegistry class. And I could find that my payload is split into each element in the part of the link below.
https://github.com/spring-cloud/spring-cloud-function/blob/v3.0.8.RELEASE/spring-cloud-function-context/src/main/java/org/springframework/cloud/function/context/config/NegotiatingMessageConverterWrapper.java#L84
However, the header information was not split into each element.
I needed the header information (for kafka key) when converting my message. However, the header contained multiple elements of information, and the payload had a single element of information, so I couldn't convert the message correctly.
Describe the solution you'd like

Solution 1 : The header of the message should contain only the header of a single element.
Solution 2 : Or, in order to know what index to refer to in the multiple elements header, the index of the message must be provided in the header.

Describe alternatives you've considered

Solution 1 : Since the message contains only the header of a single element, the custom message converter can access the specific message header correctly.
Solution 2 : Custom message converter can correctly access their own header using index.

Additional context

I am not sure I follow. Based on your description you need to convert message in some "special way" and chose to implement custom MessageConverter. We're giving you all the headers and you can do what you need to do. We are not aware of any semantics with regard to Kafka or anything else. That would have to be done by user.
Please let me know if I am missing something

I am closing it due to lack of follow up from the original reporter. Feel free to reopen it and provide more detals


dynamic() conflicts with newly introduced built-in type in OTP26-rc2

Elixir version
1.14.3
Database and Version

Ecto Versions
3.9.5
Database Adapter and Versions (postgrex, myxql, etc)

Current behavior
Ecto fails to compile with following error:
== Compilation error in file lib/ecto/query.ex ==
** (CompileError) lib/ecto/query.ex:431: type dynamic/0 is a built-in type and it cannot be redefined
    (elixir 1.14.3) lib/kernel/typespec.ex:936: Kernel.Typespec.compile_error/2
    (stdlib 5.0) lists.erl:1599: :lists.foldl_1/3
    (elixir 1.14.3) lib/kernel/typespec.ex:226: Kernel.Typespec.translate_typespecs_for_module/2

because OTP26-rc2 introduced a new built-in type dynamic().
Expected behavior
Ecto should compile 🤷♂

Pushed a fix to main and released v3.9.5.


Rewrite for single page applications

The website I want to load into a webview is designed as a single page application, is there any way I can rewrite all url requests to index.html?
With nginx I'd do somthing like
server {
listen 8080;
listen [::] 8080;
root /root/bundled;
location / {
try_files $uri $uri/ /index.html;
}
}
How can this be done with swifter?

You could use notFoundHandler.
 server.notFoundHandler = { request in
     return .ok(html("Not found :(")
   }

The feature added in #479 would allow you to do this:
server["/**"] = { request in
    return .ok(.htmlBody("You're at: \(request.path)"))
}

However, this won't match / directly, so a separate handler needs to be added for that:
server["/"] = { request in
    return .ok(.htmlBody("You got the root!"))
}


Add support for terraform language

This is based on #69, better diff: pokey/reinstate-local-web-tree-sitter-build..danlamanna/add-terraform-language.

whoops looks like this got deleted / closed when I merged #69

whoops looks like this got deleted / closed when I merged #69. Are you able to recreate the branch? There should be a button on this PR, but I don't seem to have it on my side 🤔

Nope, PRs have to be recreated unfortunately :(.


Long login time

We are faced with the problem of constant monitoring of Security Events. With a small session, there is no way to display them on a separate large screen for display.
We would like more session time in order to see all Security Events on a separate screen in real time. For example, 8 hours or longer.
Perhaps this can somehow be implemented differently, for example, a special display mode with the allocation of new non-named events.

Are you referring to UI session timeout? Maybe a different workflow, for example, using Prometheus, https://github.com/neuvector/prometheus-exporter; or Splunk app, https://splunkbase.splunk.com/app/6205/


How to configure unattended MSI install to use "write registry keys" option?

The MSI installer is capable to write some registriy keys in Windows, e.g.
HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Runtime Environment
CurrentVersion    REG_SZ    1.8
I only can enable this feature when using the GUI-Installer of the MSI file. I'm searching for the command line switch to enable this feature. Is there any docu about the cmd line installation of AdoptOpenJDK?

Take a look at https://github.com/AdoptOpenJDK/openjdk-installer/blob/master/wix/README.md#features-available
In principle you would run:
msiexec /i OpenJDK8-jdk_xxx.msi ADDLOCAL=FeatureOracleJavaSoft,<any other features>
FYI when using ADDLOCAL only the features you specify are installed so so if you still want java to be in your PATH you'll have to specify that feature too.


[BUG] Want to improve Footer

Description

want to improve the footer... plxz assign @vis
Screenshots
No response
Any additional information?
No response
What browser are you seeing the problem on?
No response
Checklist

[X] I have checked the existing issues
[X] I have read the Contributing Guidelines
[X] I'm a GSSoC'24-Extd contributor
[X] I'm a Hacktoberfest'24 contributor
[X] I am willing to work on this issue (optional)

Hello, I've fixed this issue!
This is what I've done:
To improve the footer, we'll update the CSS in the Style.css file. We'll enhance the footer's design, make it more responsive, and ensure it looks good on various screen sizes. Here are the key changes:

Redesign the footer layout
Improve color scheme and contrast
Enhance responsiveness for different screen sizes
Add hover effects to social media icons
Adjust spacing and alignment

Here's the updated footer CSS:
I also created pull request: https://github.com/vishanurag/Canvas-Editor/pull/1109

[!CAUTION]
Disclaimer: This fix was created by Latta AI and you should never merge before you check the correctness of generated code!

This bug was fixed for free by Latta AI - https://latta.ai/ourmission


Implement EIP 7742 in Prague

This PR implements EIP 7742 on top of Prague. It have done some brief tests with the execution-spec-tests PR and these changes seem to work.
~This PR should not be merged since the status of EIP 7742 is very uncertain.~
Please check that EIP 7742 is actually going into prague-devnet-5 before merging.
Cute Animal Picture

Thanks again for adding this :D
We were wondering the following should be added too:
https://github.com/spencer-tb/execution-specs/commit/07f4774873cf25e062d925f43c740f6fdf8ada24#diff-1ead0e19bcf5b72f6e0ff1f07a9d44a25a738020803d5ea42839da1e3d58a8c5R433-R434
Checking for contract creating blob txs!

Created https://github.com/petertdavies/execution-specs/pull/4 to implement changes in https://github.com/ethereum/EIPs/pull/9122.
On a side note, I'm seeing zero value in the result's field blobGasUsed, regardless of whether a blob is included in the transactions or not, I'm not sure if this is derived from the changes in this PR though @gurukamath

On a side note, I'm seeing zero value in the result's field blobGasUsed, regardless of whether a blob is included in the transactions or not, I'm not sure if this is derived from the changes in this PR though @gurukamath

Could this be the issue?


rake cconfig:info  undefined method `upcase'

Not sure if this is a bug or I am doing something wrong, but I get this error.
> rake cconfig:info

[Mailer config] Host:     portus.test.lan
[Mailer config] Protocol: http://
rake aborted!
NoMethodError: undefined method `upcase' for nil:NilClass
/Users/vadim/.rvm/gems/ruby-2.3.3/gems/cconfig-1.1.0/lib/cconfig/cconfig.rb:39:in `initialize'
/Users/vadim/.rvm/gems/ruby-2.3.3/gems/cconfig-1.1.0/lib/tasks/info.rake:29:in `new'
/Users/vadim/.rvm/gems/ruby-2.3.3/gems/cconfig-1.1.0/lib/tasks/info.rake:29:in `block (2 levels) in <top (required)>'
/Users/vadim/.rvm/gems/ruby-2.3.3/bin/ruby_executable_hooks:15:in `eval'
/Users/vadim/.rvm/gems/ruby-2.3.3/bin/ruby_executable_hooks:15:in `<main>'
Tasks: TOP => cconfig:info
(See full trace by running task with --trace)

@Vad1mo hopefully this is fixed in mssola/cconfig#5. Could you review it ? or check that it works for you ?

can do so, I first need to figure out how to build a gem and use an unpublished gem in Portus.  Sorry I am not an Ruby expert so it might take some time to figure it out.

Works now as expected. mssola/cconfig#5 fixes the problem


Let's Encrypt not working

The Let's Encrypt doesn't work. When try https, the certificate is not trusted

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ada58160-dc50-b977-d3bc-65f92406e16d
Version Independent ID: 5c596e04-fddb-effe-3b72-575f4df42d4e
Content: Configure ingress with Azure Kubernetes Service (AKS) cluster
Content Source: articles/aks/ingress.md
Service: container-service
GitHub Login: @neilpeterson
Microsoft Alias: nepeters

@tarifrr Thanks for the feedback. We are actively investigating and will get back to you soon.

@tarifrr / @vasivara-MSFT - this article has been completely updated to use cert-manager plus a few other fixes. The update will publish tomorrow morning around 10:00am PST. I would recommend checking back after this time and giving it another shot.

@neilpeterson - Whilst creating the CA cluster issuer and Certificate Objects using kubectl, do we install it on the namespace where we are deploying our application?

@neilpeterson - It's already mentioned in the article. Never Mind. Thanks!

@tarifrr - thanks for following up. If you have any other issue, please let us know.
#please-close


Updating go-restful to the latest version

Lots of great new stuff.

We now have ServiceErrorHandler which can be used for https://github.com/GoogleCloudPlatform/kubernetes/issues/4347
https://github.com/emicklei/go-restful/pull/173 fixes https://github.com/GoogleCloudPlatform/kubernetes/issues/4083
https://github.com/emicklei/go-restful/commit/358e2483dac774992c858945b509f560547c07ba should fix our problem of swagger spec rearranging each time we run hack/update-swagger-spec

cc @bgrant0607

Awesome. Build fails though.
# github.com/GoogleCloudPlatform/kubernetes/pkg/api/validation
_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/api/validation/schema.go:129: invalid operation: fieldDetails.DataTypeFields.Items[0] (type *swagger.Item does not support indexing)
!!! Error in /home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/hack/lib/golang.sh:305

Thanks! Updated the code to fix that.
Verified that hack/test-go passes on my local machine.

LGTM.


WIP/RFC: Improving the performance of fetching topics and consumer groups

I found KafkaHQ quite nice -- but it is way too slow and times out even with a "small" local setup of our environment.
Looking a bit at the trace logging and the code I found a few places that definitely could be improved:

TopicRepository was fetching all topics and then filtering them
ConsumerGroupRepository was similar, and additionally also did that when looking up offsets

The linked commits "help" (my local setup is now nice and fast to load the /:cluster/topic page), but it still times out in my test production setup with a few hundred consumer groups. Ultimately I do want this to work on my production setup, which has similar amounts of consumer groups, but considerably more topics distributed over these groups.

Thanks @ankon for this PR.
Will have a look soon can be a good optimization but I need to look at closer, since you change how you get all consumer groups for a topic.
As I see, I don't see any other way to list all consumer group to have all the group subscribe to a topic .
Maybe this PR will me make think there is a better way.
Maybe there is a related issue on perf #55 with cache that could help you on response time ?
Will give you my feedback soon

[...] change how you get all consumers groups for a topic.

True, I do change how to do that -- but that change shouldn't be externally visible, it merely reorders calls so that one avoids querying kafka too often, and rather use bulk queries.
Especially the ConsumerGroupRepository::findByName was a problem, because it would first find all consumers groups, then do a query for each of these, which produces a bunch of consumed topics for that group as a side-effect. And then it would query information about these topics in that loop, which requires creating and closing a Kafka consumer (~3s depending on the heartbeat times for us). In our case at least the consumer groups actually use the same topics in many cases, so the refactoring uses that to first build a complete list of all topics used by any consumer group, and then just do a single query for all of these.

Maybe there is a related issue on perf #55 with cache that could help you on response time ?

Caching is certainly an interesting approach as well, and I'll try to have a look at that cache. Still these would be complementary: One improves the performance for the querying, and the caching reduces the number of times you'd have to query in the first place I suppose.

Hello @ankon .
I've a quick look but I will need to look at deeper.
My first observation: It seems that yours changes is "breaking" pagination on topic list:

To be more precise, the order is not the same (natural sort order on dev branch, seems to be random on your branch).
Also .map(CompletableFuture::completedFuture) : break the lazy loading on pagination, there is some topic that are loaded that is not displayed on current page.

To have more debug output, try to run this :
curl -i -X POST -H "Content-Type: application/json" \
       -d '{ "configuredLevel": "TRACE" }' \
       http://localhost:8080/loggers/org.kafkahq

or change logback.xml.
And change the pagination with kafkahq.topic.page-size in order to see pagination in action.
The most important part is to have only only topic on current page that is loaded, on big cluster describe topic offset can be very long, and pagination are here to avoid loading all the topic information.
Tell me what do you think and thanks for the work :+1:

Any progress on this PR? We have similar performance issue with a cluster having many topics and consumer groups. We are really interested with these enhancements.

Thanks for taking a look!

To be more precise, the order is not the same (natural sort order on dev branch, seems to be random on your branch).

Fair point, I likely lost that by joining things together. Fixed that by an explicit sorting step now, and while there rebased the branch onto the current HEAD of dev.

The most important part is to have only only topic on current page that is loaded, on big cluster describe topic offset can be very long, and pagination are here to avoid loading all the topic information.

Right, sensible. I now configured my own instance with a low enough page-size, and then started to have a look. Reading through org.kafkahq.utils.CompletablePaged it seems though that pagination so far works in quite a roundabout way: The TopicController creates a bunch of futures for all matching topics, and the CompletablePage then cancels all but the needed futures. By moving towards more "batching" of requests I did indeed make the paging less efficient in how it reduces work (to the point where we have all results ready before even looking at the pages), but at the same time I also avoid most of the roundtrip work that the paging would cancel.
I need to think a bit further and read through more code here first, it's quite possible that the batching approach helps a lot in my situation (many groups, few topics), and would produce the complete opposite effect in other cases.
@ftardif Sorry for the delays, this is right now "next to" my day job :) I do try to keep the branch compiling though, and for local testing you could try to checkout this branch and deploy it into your own docker registry (or use the produced kafkahq jar). Feedback whether this improves the world for you would be very much appreciated!
npm install && npm run build && /gradlew test shadowJar && cp build/libs/kafkahq-*.jar docker/app/kafkahq.jar && docker build . -t kafkahq:$(git describe --abbrev --tags --dirty)

I need to think a bit further and read through more code here first, it's quite possible that the batching approach helps a lot in my situation (many groups, few topics), and would produce the complete opposite effect in other cases.

I've now opted for the most "direct" approach: List the topics (or consumer groups) first, then page over that, and then lookup the content from Kafka for only that page. Depending on how your groups/topics distribute over each other this may be more or less "good", so another round of review/testing would definitely be appreciated.

Thanks @ankon.
To be honest, I'll try to explain you what the last research let me understand recently with espacially  #55, #137 and some others reports.
I never be able to reproduce the latency on my side because mostly my kafka cluster is local (on local network).
I've some chance to look at KafkaHQ on Confluent Cloud and Aiven (on internet) with more topic & consumer groups.
Here is the history :

before the TopicRepository was doing exactly what you are doing (send batch query with all the topic & consumer group in one time, and this time there is no pagination), result better performance than now on small cluster.
after some report, I've seen that people with a lot of topic / consumer group report a real latency, so I decided to do the version with CompletableFuture, result, only load current page but remove the batch fetch, see a better performance on my local computer. What i just realized, is that default executor will be ForkJoinPool (with 1 worker per cpu). Since most of people used KafkaHQ in docker with a limited CPU  (mostly 1 from report), that will completely ruin the uses of CompletableFuture allOf, all with be done in sync and not in parallel like I hope.

one solution here will be to provide a custom ExecutorPool configurable to have some more parallel query (I still think about provide this option as a quick & dirty solution).

I'll try to add some cache on #55, but it's clearly a thing I don't want at start of KafkaHQ: Yahoo Kafka manager is not real time with a lot of cache that is really frustrating when developing new feature based on kafka.

But some report like #137 show what is the main trouble in the main conception of these page :

getOffset of topic can be very long, especially on a big cluster or unstable cluster.
Similarly, display the consumer group (with lag) on the same page can be slow.

So what will be next action IMO :

Move a lot of thing to async ajax call espacially offset / log dir size and consumer group on topic page.
Move list of topic lag to async on consumer group page.
let people customize the number of parallel thread to complete the CompletableFuture on a page.

I'll try too long to avoid this but since the consumer offset query can lead to a timeout on a lot of case, there is no way IMO to avoid this.
Since this is a big work, I'll try to review the last addition on your PR next week to see if It will handle some improvement that can be useful in a short time before big refactoring to move all async.
Thanks for your work and keep in touch next week :+1:

Hey @here !
New release of version 0.10.0 include the first easy part :

let people customize the number of parallel thread to complete the CompletableFuture on a page.

https://github.com/tchiotludo/kafkahq#pagination with option kafkahq.pagination.threads.
Tell me what is your feeling and if it help in your case.

I'll try to rebase the branch onto that.
I'm a bit suspicious here, as "more threads, more better" hasn't always been true in my experience :)
FWIW, what I now did to make KafkaHQ be somewhat useable: I disabled the fetching of consumer groups completely (https://github.com/tchiotludo/kafkahq/pull/134/commits/ea6ef98b5059ffa8033ea132d9e587b5013a398d), as in my case this data isn't really needed.

I've seen a better experience on my side.
But it always depend on Kafka server response time.
I've seen your last modification, seems to be clearly what I state on my previous comment, move to async will do the trick. Just need to find time

Interested in this, our cluster has 20 topics and it takes 10 seconds to load the topic list page (pagination =  10 records so not even all our loaded).

@jorgheymans even with last version  ?

@tchiotludo yes that's with the latest version, we just started using kafkahq :)

FWIW, we also don't need need consumer group fetching, group lag is kept track of using prometheus metrics. So what i was thinking as possibilities:

have a config toggle that allows sites to disable it if they have no use for it. This should be easy and not impact existing users.
instead of showing consumer groups for all topics by default, have a button or something in that row cell that upon click fetches the group info only for that topic. I don't think there is a need to have a global overview of all consumer groups for all topics, and if so that should be made a different screen alltogether instead of packing it all in the topic list.

WDYT ?

To be honest, I really think the consumer group lag is a major feature in KafkaHQ. I'm using personally every day.
This feature slow down the application for now, and the only solution is to go async IMO.
It will help both people, people that need this information and people that don't need it since the application will be very quick to display main information.
This async feature will took more time but will resolve all the issue about performance (#55) since this is not the only things that take time on topic list (offset also take time), so removing consumer groups is only a quick patch that will not resolved all the performance issue.
Another options is experimental cache : https://github.com/tchiotludo/kafkahq/issues/55#issuecomment-530553103
It will speed up the whole KafkaHQ experience but that can lead to some bug.

OK i understand it's an important feature. I was just looking for a quick way out i guess :-)
Also in the case where you have let's say 100 consumer groups for a topic would it clutter the table or it's graceful enough to handle this ? Consumer groups accumulate over time, with applications / console consumers / kafkacat etc etc .

Good point, Never happen for me with like 4-5 consumers groups per topic & console-consumer being deleted. Will keep it in mind when I will do async

Having consumer group information would certainly be nice for us ... to some extent.
We have a multi-tenant solution, with a couple of topics per tenant. Multiple services (>40!) are working over these topics, and each of these services for every instance creates a unique consumer group. On our test environment I just checked now: We have maybe 5 tenants, and right now almost 4000 consumer groups. Most of these groups are "dead", and their lag would be irrelevant (it roughly would indicate how long these groups are dead :D).
Now, this may be fairly unique, and it may be "wrong" in terms of best practices. We're considering changes that would reduce the number of groups, but realistically the number will always be a lot higher than just 5 per topic. At that point the UI of KafkaHQ isn't able to show the groups anyways apart by cutting of some -- and there isn't a good heuristic really to work out which ones to cut: Drop the ones with a high lag would be fine for us, but would obviously miss to show the ones with a high lag not caused by the group being dead. :)
I think a switch to disable this actually works quite well in this case. I have now been able to use KafkaHQ for diagnosing some support questions, and I love it even without the consumer groups :)

We have a cluster with hundreds of topics and hundreds of consumer-groups.
Latency due to calculating consumer lag on HQ is the #1 pain for the dev
team. I would very much agree with a toggle switch to keep the topic and
consumer page leaner and only join the consumer lag information once we
double click on either a specific topic or a consumer group.
thanks
Frederic Tardif
Le lun. 25 nov. 2019, à 09 h 32, Andreas Kohn<EMAIL_ADDRESS>a
écrit :

Having consumer group information would certainly be nice for us ... to
some extent.
We have a multi-tenant solution, with a couple of topics per tenant.
Multiple services (>40!) are working over these topics, and each of these
services for every instance creates a unique consumer group. On our
test environment I just checked now: We have maybe 5 tenants, and right now
almost 4000 consumer groups. Most of these groups are "dead", and their lag
would be irrelevant (it roughly would indicate how long these groups are
dead :D).
Now, this may be fairly unique, and it may be "wrong" in terms of best
practices. We're considering changes that would reduce the number of
groups, but realistically the number will always be a lot higher than just
5 per topic. At that point the UI of KafkaHQ isn't able to show the groups
anyways apart by cutting of some -- and there isn't a good heuristic really
to work out which ones to cut: Drop the ones with a high lag would be fine
for us, but would obviously miss to show the ones with a high lag not
caused by the group being dead. :)
I think a switch to disable this actually works quite well in this case. I
have now been able to use KafkaHQ for diagnosing some support
questions, and I love it even without the consumer groups :)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tchiotludo/kafkahq/pull/134?email_source=notifications&email_token=AABWCFS3Z42HTUWRPBOGIHTQVPOXXA5CNFSM4JDRZWE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFCS2XY#issuecomment-558181727,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABWCFTDC4CFAZIX2TLNABTQVPOXXANCNFSM4JDRZWEQ
.

As i say, this is not the only things that is slow, removing consumer groups will not save you for others things (like topic offset that is the main slowness reason).
Async is the only solution:

removing all the offset fetch / consumergroups from main page.
maybe allow cache only on this part (that is not needed in real time).

PR are welcome guys :smile:

@tchiotludo i would provide a PR but #153 :/

@ankon maybe https://github.com/tchiotludo/kafkahq/commit/ea6ef98b5059ffa8033ea132d9e587b5013a398d could be part of a separate PR so we can contribute to this change only ?

Sure, we can do that -- after all this is a bit the point of this PR :)
See #159, note that I did only compile-test that version.

hey @ankon, I just push a new dev version that will "I hope" fix a lot of performance issue.
So first sorry for that, I started with a completely different approach than you, and after seen some good things on this PR but when I try to merge it, there is too much conflict, and I "stole" some part of this PR, really sorry for that.
Maybe you can try with dev version to see if it fit with you're need and maybe a new PR with all addition I don't see ?
I know this is hard work for you, so as you wish and sorry to have stolen some parts without merging it :pray:

Thanks for your work!

So first sorry for that, I started with a completely different approach than you, and after seen some good things on this PR but when I try to merge it, there is too much conflict, and I "stole" some part of this PR, really sorry for that.

No problem at all: This PR wasn't really meant for merging as-is, but more for discussion -- and I think we did have that :)

Maybe you can try with dev version to see if it fit with you're need and maybe a new PR with all addition I don't see ?

I actually just deployed it, as I wanted to try a "unmodified version" to see whether we still would need additional patches and it would make sense to start rebasing/merging. Turns out: We don't, the version 10ba9ad707baec4d285ff98b1fed4353c48dde7 seems to work just fine in our environment. We did still keep the consumer groups disabled, because we don't need them anyways (see above).
So I think maybe best could be to close this PR, and if there is new/more/etc things open a new one. WDYT?

Thanks @ankon , really glad to hear that this version is Ok for you :)
Mean that we are on the right way and first part of optimization is working.
I'll release a new version really soon :tada:
Need to take some more time to go async for all time consuming operation (allowing to remove the remove of consumergroup cols) but will be more longer.
Will wait for more community feedback with current version.
So like you said I will close this one and feel free to add a new one if you have more optimization in mind, thanks :+1:


fix: bug 1675 - UX: shove error panel close button flush right

UX guidelines have the close buttons flush right

:tada: This PR is included in version 0.324.26 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


[BUG]: show/hide model description is broken for short text

Describe the bug
Models with short descriptions are overlapped by the button.
It also doesn't make sense to show an expand/collapse button at all when a description is that short.
To Reproduce
Steps to reproduce the behavior:

Open a model with a short description (perhaps Zhao2020? As shown in the screenshot below)
Notice that the beginning of the text is overlapped.

Expected behavior
Don't show the faded background or expand/collapse button when the description is less than 480 characters or so.
The description text is set to wrap after 120ch (approximately but not exactly 120 characters). The button allows 3rem (about 2 and a half lines, depending on line height) of uninterrupted description above it before the fade begins.
Be sure to test with many different lengths of text, with different window sizes.
Screenshots

Additional context
We expect to use similar behaviour and appearance in other parts of the app. Consider extracting a new expandable text component for later reuse.

No longer relevant due to design changes.


Airflow importers don't match API

When attempting to interact with the API, using the dict returned by Scene.to_dict() as a payload, I received the following errors from the API (I fixed the first one by manually editing the payload, and then received the second):

"The request content was malformed:\nObject is missing required member 'metadataFiles'"
"The request content was malformed:\nObject is missing required member 'resolutionMeters'"

Duplicate on #582 part 3


Group depends for watched deps within a class

Supposed to fix #415...but let's see if the existing test suite passes first...
I probably need a review from @philippjfr at this point before I go further, as 1. I'm a bit out of practice ;) and 2. seemingly if I ever try to inspect anything param-ish using pdb, I get Fatal Python error: Cannot recover from stack overflow and a core dump :( (no doubt related to the ever recurring repr problems).
To do:

[ ] add tests

Coverage increased (+0.02%) to 79.469% when pulling 3867a702714bd639670e66460c4d7e1041c44db9 on grouped_depends into a1adb2f27fb1b29829df0345feb771196e531e63 on master.

I bumped into this issue while working on the new Event parameter:

The tests have passed and I would like to see this merged! At a glance the code looks good but I haven't looked closely enough yet..

@jlstevens I guess you will add a test and finish this off? (I'm very happy with that if so - thanks :) )

@philippjfr @ceball Jim and I agree to merge now but please do a post review merge and view the comments (I'm now going to test this on another PR).

As an aside, note similarities to code elsewhere that I was using as a basis for this change:
https://github.com/holoviz/param/blob/301b9617e84f3578364393d5ae30c452d1c19cd0/param/parameterized.py#L368-L377
(i.e. string specs vs not - the separation makes it difficult to remember - just saying/not that I am going to implement a solution to that)

Looks like it is working now:

The only thing I am wondering is whether you might expect the callbacks to execute in definition order (at least in Python 3 where things are generally ordered) i.e 'A' then 'B' then 'C'. I don't think we ever promised such semantics but it might be nice if it is easy to provide...


Use git-lfs for the models?

It would be neat to use git-lfs to put the models in the repo, instead of a bunch of 50MB zips downloaded separately.

Some git services are unstable in China, though git-lfs is a good idea.


SPIKE: Review the process of removing users from the two GH Orgs

Background
When removing users from the MoJ and GH Org, the user needs to be removed from both Orgs to recover the license. If a GH Owner removes the user from one Org we need to be informed or that person needs to remove the user from both Orgs. There is the leavers process that informs the ops-eng team but not sure if that is being used by users in the AP Org. If not action is taken once the user is dormant for 90 days the inactive repo automation will pick that user up and should remove the user.
MoJ GH Owners
AP GH Owners
We need to know what the Data dept / AP / AS process for when a user leaves the MoJ to make it align with our process
The leavers process form: https://leavers.form.service.justice.gov.uk/
On the leavers form: "For D&A, I’m not sure what to select on that. D&A falls under HQ, but not Central Digital, Engineering, SSIGG or Strategy and Operations as far as I know"
Maybe the leavers form needs updating as well?
Maybe speak with Calum Barnett to see what their process is / spread the word to use the leavers form

Example where a user has left the Org but no one has informed us and the user still has access to the tooling.

Closing this story. At present the is no JML process for the AS Org. We have to rely on the automation at the moment for ensuring users are removed from the org.


Replace set/add with cmov in secp256k1_gej_add_ge.

Use a conditional move of the same kind we use for the affine points
in the storage  type instead of multiplying  with the infinity flag
and adding.  This results in fewer constructions to worry about for
sidechannel behavior.
It also might be faster: It doesn't appear to benchmark as slower for
me at least; but I think  the CMOV is faster than the mul_int + add,
but slower than the set+add;  making it a wash.

I also tried adding a fe_add_int (e.g. r[0]+=a;) that made do strictly no more work, but I couldn't benchmark it as any faster; and it didn't seem worth the additional code to review.  ... but inlined it should save a fair chunk of code size (one instruction vs ... many).
@sipa if you want I can post the trivial code for that.

I measure a gej_add_ge slowdown of 0.9%, and a signing slowdown of 0.2% or so. ACK.


docs: add api scope to dev tunnel docs

Changes

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#747  👈
#740
#743
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @AngelOnFira and the rest of your teammates on  Graphite

Converting to draft since it depends on drafts


Plugin config data use  by plugin_handler  template in soul-admin

soul-admin-->plugin-->edit
Please read the Contribution Guideline before submitting the PR

111

get


Decouples light-sync state from chain spec

This decouples the light-sync state from chain spec. First, the
light-sync state currently only works with BABE+Grandpa, so not
all Substrate based chains can use this feature. The next problem was
also that this pulled the sc-consensus-babe and sc-finality-grandpa
crate into sc-chain-spec.
If a chain now wants to support the light-sync state, it needs to add
the LightSyncStateExtension to the chain spec as an extension. This is
documented in the crate level docs of sc-sync-state-rpc. If this
extension is not available, SyncStateRpc fails at initialization.
polkadot companion: https://github.com/paritytech/polkadot/pull/3568
Fixes: https://github.com/paritytech/substrate/issues/8886

What is the "light sync state" used for? Looks like the code that was supposed to use it was never written? Why not just remove it?
Also, the light client does not really need to have epochs and authorities in the chain spec. All that's needed is a trusted finalized block hash. The rest should be recoverable from the state.

What is the "light sync state" used for? Looks like the code that was supposed to use it was never written? Why not just remove it?
Also, the light client does not really need to have epochs and authorities in the chain spec. All that's needed is a trusted finalized block hash. The rest should be recoverable from the state.

It used by the light clients as a starting state. @tomaka should be better at explaining for what this is actually required.

It's used by smoldot to start syncing from this checkpoint, on substrate only the code to export the sync state exists. This was written before warp sync was implemented on smoldot so maybe it's not super relevant anymore.

Also, the light client does not really need to have epochs and authorities in the chain spec. All that's needed is a trusted finalized block hash. The rest should be recoverable from the state.

I think when this was written there was no easy way to get this data from the runtime, I agree that now if we still want to have a checkpoint for light clients to start syncing we only need to provide finalized block hash, we can even start warp syncing from such a block. Since this is used by smoldot currently I'll defer to @tomaka on what to do right now.

The light sync state was "designed" and added before Grandpa warp sync existed.
At the time, we were also trying as much as possible to avoid having to do runtime calls in the light client (as this makes the light client binary size much larger than otherwise, and requires downloading the runtime code from the network). For this reason, retrieving the Babe epoch and finalized authorities from the state wasn't really a good solution.
Nowadays, yes we probably just need a finalized block hash. However, knowing the list of Grandpa authorities of this finalized block is necessary in order to verify the warp syncing. This list could also be retrieved from the network, but this adds some latency and most importantly we run the risk that all the nodes of the network have already pruned the block and are incapable of giving us that list.

So, you would like to keep it @tomaka?

Right now, this light sync state contains what a light client needs in order to start syncing from a certain block.
We could remove the Babe-related stuff from the light sync state, but keep the Grandpa-related stuff, in which case the light sync state would contain what a light client needs in order to warp sync from a certain block.
If we remove also the Grandpa-related stuff and keep only a finalized block header or hash, then we risk that everyone on the network has pruned the block, and/or that we make lots of useless network requests trying to find a node that hasn't. This is too much removal IMO.
Smoldot can accommodate any of the three, I don't care, but I'd be leaning towards removing the Babe stuff.

Okay, then I'm going with keeping it as is for now.

bot merge

Could it be that a Cumulus companion is missing for this?


partitionsort Fix DCHECK failure

Proposed changes
In partition sort operator, blocks vector in value place will be clear after a batch data. If we sink data with eos after this clear, DCHECK will cause a illegal memory access to empty vector.

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


Update get-all-vulnerabilities.md

Updated API description with odata filter queries

Issue#https://github.com/MicrosoftDocs/microsoft-365-docs/issues/5933

@JohanFreelancer9 Please copy review

@siosulli Please merge this PR. Thanks

@siosulli Gentle reminder


Add blank_search_value option

I have a use-case where I'd like there to be a default search if my search is blank. I've added this feature to a fork here (search blank_search_value).
Unfortunately this fork is out of sync with yours, so I can't directly contribute a pull request. Would it be possible to add this feature?
Here is the doc you can add to the readme:

blank_search_value: str = None

If not None, will set the default search value when the box is initialized or reset.

In theory, your default option should work, but in practice when I click on the searchbox, the default search doesn't show up.
Furthermore, if I reset or clear the box the default search does not show up again.

Hey, just for my understanding: this is not about the default that is returned in python but the text that is shown in the searchbox? So setting something like the initial search term?

Hey, just for my understanding: this is not about the default that is returned in python but the text that is shown in the searchbox? So setting something like the initial search term?

Exactly, yes. Anytime the box is blank, use this default search term.
My use case here is a folder search combobox. When the box is blank, I want it to show all files in a folder. I don't want users to have to type "." to see everything.

Could you also use default_options for this? Like:
result = st_searchbox(
    search,
    default="default_folder_return",
    default_options=["your", "list", "of", "folders"],
)

The problem with this is that it won't actively run the search again, which may change over time.
Also, that wasn't working for me when I clear the result.

I see the issue, just hesitant to add an additional argument that might be covered by already existing ones & the state updates having changed somewhat since the fork you linked above..
Can you maybe check if this https://github.com/m-wrzr/streamlit-searchbox/pull/61 would cover the usecase? this resets the searchbox with also updating the default_options when using clear_on_submit = True
pip install -i https://test.pypi.org/simple/ streamlit-searchbox==0.1.17a

Works perfectly! Thanks for helping with all my issues. This component is excellent and VERY usable for extreme use-cases. I might have one more low-priority minor feature request ;)


Illegal instruction error when running on server(centos7)

https://github.com/facebook/rocksdb/issues/5912

https://github.com/OpenAtomFoundation/pika/pull/1812


Compound app UI - The buttons are too big and close toguether

Original ticket:
https://app.zenhub.com/workspaces/safe-multisig---web-5ce554debb310a35b2f8b6f8/issues/gnosis/safe-react/1580
Env: https://pr1937--safereact.review.gnosisdev.com/rinkeby/app/
Snapshot with the issue and how it should look

The compound app is super bare bone. I think it's not worth it to spend any effort to fix the UI at this point :)


[Dockerfile] Create every machine level variable as process level variable on Windows

For reviewers
If you use os.LookupEnv or os.GetEnv you might be affected.
What does this PR do?
This PR removes an env variable "filter" on Windows.
Strictly speaking, it is not an env variable filter. We just do not replicate the machine level env variables.

Motivation
We could not fetch the ECS metadata endpoint because the agent didn't have access to the right env variable.
It looks like many others are affected.

Additional Notes
N/A

Possible Drawbacks / Trade-offs
We need to make sure every code maintainer using env variable is aware of this change.

Describe how to test/QA your changes
Deploy the rc on ECS Windows. Make sure tags are fetched.

Reviewer's Checklist

[ ] If known, an appropriate milestone has been selected; otherwise the Triage milestone is set.
[ ] Use the major_change label if your change either has a major impact on the code base, is impacting multiple teams or is changing important well-established internals of the Agent. This label will be use during QA to make sure each team pay extra attention to the changed behavior. For any customer facing change use a releasenote.
[ ] A release note has been added or the changelog/no-changelog label has been applied.
[ ] Changed code has automated tests for its functionality.
[ ] Adequate QA/testing plan information is provided if the qa/skip-qa label is not applied.
[ ] At least one team/.. label has been applied, indicating the team(s) that should QA this change.
[ ] If applicable, docs team has been notified or an issue has been opened on the documentation repo.
[ ] If applicable, the need-change/operator and need-change/helm labels have been applied.
[ ] If applicable, the k8s/<min-version> label, indicating the lowest Kubernetes version compatible with this feature.
[ ] If applicable, the config template has been updated.

Can you please remove "ping" commits from the PR? (It wouldn't remove reviewers, just to keep the history clean).
Would it make sense to modify the filter to include the variables needed for ECS instead of removing filter altogether?
And the QA should probably include checks on non-ECS Windows too.

Can you please remove "ping" commits from the PR? (It wouldn't remove reviewers, just to keep the history clean).

It should be done.

Would it make sense to modify the filter to include the variables needed for ECS instead of removing filter altogether?

It is still unclear what this filter does. I think we do not have anything similar on linux and we think this filter can possibly cause a lot of hard to detect bugs.

And the QA should probably include checks on non-ECS Windows too.
Indeed.


Butto to start in AR without needing the 3d view on the website

I'd like to know if it's possible to have a button to launch the AR view without the need to have the 3d view that the model viewer offers by default, all I want is a basic button, once clicked the AR mode is launched, I don't want the 3d viewer viewing my 3d model on the web page.
Thanks

See #4315.


API usage issues - TOP being ignored, timeout, etc.

using version azure-mgmt-consumption==9.0.0
The API .usage_details.list(
scope=f'/providers/Microsoft.Billing/billingAccounts/{billingAccountIdLong}/billingProfiles/{billingProfileId}'
,metric='amortizedcost',
top=100)
Completely ignores the top clause, also if you use it '100'.
The API also throws (429) Too many requests. Please retry after 60 seconds.
when you try to consume it.
Could you please advise on what is the correct way to use this API to get the current months all charges from Azure, to all subscriptions in our billing account / billing profile?
Is there a way to govern to get back e.g. last 3 days only?
The closest we get if we start to loop over each subscription separately - but this both seems a bad approach, as we have billing profiles and consolidated accounts - while also it is VERY slow - 6 accounts data jan 1 to 15 takes 13 minutes, and these are not even our big accounts.
Thank you in advance

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 0e8c4735-1579-58a0-f375-096d5dbdc2c4
Version Independent ID: 30a48745-054c-2748-f326-ab3b498dc144
Content: azure.mgmt.consumption.operations.UsageDetailsOperations class
Content Source: docs-ref-autogen/azure-mgmt-consumption/azure.mgmt.consumption.operations.UsageDetailsOperations.yml
GitHub Login: @lmazuel
Microsoft Alias: lmazuel

@msyyc can you take a look?

Hello， @OTDani ，I tried to solve your problem. Here is the code. I hope it will help you
import os
import datetime

from azure.identity import DefaultAzureCredential
from azure.mgmt.consumption import ConsumptionManagementClient

credentials = DefaultAzureCredential()
subscription = os.getenv('SUBSCRIPTION_ID')

consumption_client = ConsumptionManagementClient(DefaultAzureCredential(), subscription_id=subscription)
billing_account_id = 'your billing account id'

#Get today's date and the date three days ago
current_date = datetime.datetime.now()
offset = datetime.timedelta(days=-3)
past_date = (current_date + offset).strftime('%Y-%m-%dT%H:%M:%SZ')
filter_expression = "{} and {}".format(current_date.strftime('%Y-%m-%dT%H:%M:%SZ'), past_date)

res=consumption_client.usage_details.list(
    scope=f'/providers/Microsoft.Billing/billingAccounts/{billing_account_id}/billingProfiles/{billing_account_id}'
    ,metric='amortizedcost',
    top=100,filter=filter_expression)

We are waiting on @ccmbpxpcrew
Please let me know what is the information you need from me?

Thanks @BigCat20196 @kazrael2119 !!

in the @kazrael2119 code the scope is

scope=f'/providers/Microsoft.Billing/billingAccounts/{billing_account_id}/billingProfiles/{billing_account_id}'

Could you please double check this?  As AFAIK it should be a billing profile id at the end, this throws an error 500.

Using what i think was intended, the code times out with (429) Too many requests. Please retry after 60 seconds.

scope=f'/providers/Microsoft.Billing/billingAccounts/{billingAccountIdLong}/billingProfiles/{billingProfileId}'

but it runs, but not fine for just a subscription scope:
scope=f'/subscriptions/{azu_subscription_id}/'

Returns 1 small account taking over a minute - 4474 elements - (I'm sitting on gigabit net)
It completely ignores the date filter.
Expected data from Past 3 days, actual: the API gives back data from month's start as it would without the filter 😢

Back to the other two issues:

How to overcome  (429) Too many requests. Please retry after 60 seconds. ?
How to speed this up?

Hi @OTDani the team is going to look at it soon. I'll create a backlog item for this

Hi @OTDani , Python SDK is just tools to send/receive request/response. According to the context, it seems that server does not take the filter effect. Please follow https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/debug_guide.md to provide more detailed logs. With the logs, you could see the raw request and response from server. If the request is right with filters while server still return more data, it proves that there may be problems in server and you could make issue in https://github.com/Azure/azure-rest-api-specs/issues


Repetition fields

Currently, there is no way to obtain repetition fields beyond the first repetition.  This functionality would be incredibly helpful for a project I'm currently working on.  I noticed there is a PR here https://github.com/soliantconsulting/fm-odata-client/pull/3 which addresses this same issue, but it seems to have died after the changes requested were posted.  Is there any way this functionality could be implemented?  Or I mean, I myself could probably work on the changes that were requested from the mentioned PR.

Feel free to fork that PR and create a new one!


README error in Configuration section

Bug description
In the README file, under the Configuration section, you should change

Logstash Pipeline in main.conf at ./elasticsearch/pipeline/main.conf.

to

Logstash Pipeline in main.conf at ./logstash/pipeline/main.conf.

@enstit Thanks, I'll make sure to fix it on the next release, PRs are welcome too 😉


Unable to add generated certificate to Firefox as trusted CA

Hi, first of all, thanks for the very useful package!

Due to Firefox having more strict rules for (self signed) certificates, the generated files cannot be used to tell Firefox to trust the certificate.
I've been trying to test fixes according to this StackOverflow Post. If I have a setup which works, I'll edit this post.
Is this something you'd consider supporting, or is it out-of-scope?

I got it working with the following changes (vendor/ryoluo/sail-ssl/nginx/generate-ssl-cert.sh):
#!/bin/sh

ME=$(basename $0)
+ROOT_KEY=/etc/nginx/certs/root-ca.key
+ROOT_CERT=/etc/nginx/certs/root-ca.crt
+CSR=/etc/nginx/certs/server.csr
KEY=/etc/nginx/certs/server.key
CERT=/etc/nginx/certs/server.pem
CN=$SSL_DOMAIN
SAN=$SSL_ALT_NAME

if [ -f $KEY ] && [ -f $CERT ]; then
    echo "$ME: Server certificate already exists, do nothing."

fi

Then calling the command below, which copies the CA certificate to your local machine
sail cp nginx:/etc/nginx/certs/root-ca.crt .

Then adding the crt file to firefox:
Firefox → Settings → Privacy & Security → Security → View Certificates → Authorities → Import

Actually, this solves the same problem as: https://github.com/ryoluo/sail-ssl/pull/27 - just in a different way. Either would be awesome if they could be supported 🙂

@ryoluo Hi, sorry for ping- but is this something you'd implement? I wouldn't mind making a PR for it.


Add StyleCop packages to the example and template projects

Same as we do for source projects we should add the StyleCop.Analyzers NuGet package to the example and template projects.

Fixed templates in #3970 and templates in #3964