2 GPUs with TP=2 run Lora inference, one GPU

2xV100上运行 vicuna-13b + lora， tp=2, 启动后如下所示，这个 gpu1 上 utility = 100% 是正常的吗？

我测试过程中，将client程序强行中断，服务中出现如下问题情况：

[uvicorn.error][2024-03-28 07:32:57][9079][MainThread][/home/xxxx/.local/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py:run_asgi():434][ERROR] Exception in ASGI application
Traceback (most recent call last):
  File "/home/xxxx/.local/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 429, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/home/xxxx/.local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/home/xxxx/.local/lib/python3.10/site-packages/uvicorn/middleware/message_logger.py", line 86, in __call__
    raise exc from None
  File "/home/xxxx/.local/lib/python3.10/site-packages/uvicorn/middleware/message_logger.py", line 82, in __call__
    await self.app(scope, inner_receive, inner_send)
  File "/home/xxxx/.local/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/home/xxxx/.local/lib/python3.10/site-packages/starlette/applications.py", line 116, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/xxxx/.local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/home/xxxx/.local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 83, in __call__
    await self.app(scope, receive, send)
  File "/home/xxxx/.local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/home/xxxx/.local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 44, in wrapped_app
    await app(scope, receive, sender)
  File "/home/xxxx/.local/lib/python3.10/site-packages/starlette/routing.py", line 746, in __call__
    await route.handle(scope, receive, send)
  File "/home/xxxx/.local/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/home/xxxx/.local/lib/python3.10/site-packages/starlette/routing.py", line 75, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/home/xxxx/.local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 44, in wrapped_app
    await app(scope, receive, sender)
  File "/home/xxxx/.local/lib/python3.10/site-packages/starlette/routing.py", line 70, in app
    response = await func(request)
  File "/home/xxxx/.local/lib/python3.10/site-packages/fastapi/routing.py", line 294, in app
    raw_response = await run_endpoint_function(
  File "/home/xxxx/.local/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/home/xxxx/.local/lib/python3.10/site-packages/maga_transformer/server/inference_app.py", line 103, in inference
    return await self.inference_server.inference(req, raw_request)
  File "/home/xxxx/.local/lib/python3.10/site-packages/maga_transformer/server/inference_server.py", line 161, in inference
    return await self._infer_wrap(req, raw_request, generate_call)
  File "/home/xxxx/.local/lib/python3.10/site-packages/maga_transformer/server/inference_server.py", line 166, in _infer_wrap
    rep = await self._infer_impl(req, id, raw_request, generate_call)
  File "/home/xxxx/.local/lib/python3.10/site-packages/maga_transformer/server/inference_server.py", line 264, in _infer_impl
    raise asyncio.CancelledError("client disconnects")
asyncio.exceptions.CancelledError: client disconnects

这个是有影响的吗？

是持续这个状态吗 一般来说占用率分别为0%、100%是由于在同步数据，持续卡在这个状态可能是同步失败，如果可以正常输出则是正常现象；
没有 这个是正常通信断开吧

2xV100上运行 vicuna-13b + lora， tp=2, 启动后如下所示，这个 gpu1 上 utility = 100% 是正常的吗？

我测试过程中，将client程序强行中断，服务中出现如下问题情况：

[uvicorn.error][2024-03-28 07:32:57][9079][MainThread][/home/xxxx/.local/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py:run_asgi():434][ERROR] Exception in ASGI application
Traceback (most recent call last):
  File "/home/xxxx/.local/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 429, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/home/xxxx/.local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/home/xxxx/.local/lib/python3.10/site-packages/uvicorn/middleware/message_logger.py", line 86, in __call__
    raise exc from None
  File "/home/xxxx/.local/lib/python3.10/site-packages/uvicorn/middleware/message_logger.py", line 82, in __call__
    await self.app(scope, inner_receive, inner_send)
  File "/home/xxxx/.local/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/home/xxxx/.local/lib/python3.10/site-packages/starlette/applications.py", line 116, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/xxxx/.local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/home/xxxx/.local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 83, in __call__
    await self.app(scope, receive, send)
  File "/home/xxxx/.local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/home/xxxx/.local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 44, in wrapped_app
    await app(scope, receive, sender)
  File "/home/xxxx/.local/lib/python3.10/site-packages/starlette/routing.py", line 746, in __call__
    await route.handle(scope, receive, send)
  File "/home/xxxx/.local/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/home/xxxx/.local/lib/python3.10/site-packages/starlette/routing.py", line 75, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/home/xxxx/.local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 44, in wrapped_app
    await app(scope, receive, sender)
  File "/home/xxxx/.local/lib/python3.10/site-packages/starlette/routing.py", line 70, in app
    response = await func(request)
  File "/home/xxxx/.local/lib/python3.10/site-packages/fastapi/routing.py", line 294, in app
    raw_response = await run_endpoint_function(
  File "/home/xxxx/.local/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/home/xxxx/.local/lib/python3.10/site-packages/maga_transformer/server/inference_app.py", line 103, in inference
    return await self.inference_server.inference(req, raw_request)
  File "/home/xxxx/.local/lib/python3.10/site-packages/maga_transformer/server/inference_server.py", line 161, in inference
    return await self._infer_wrap(req, raw_request, generate_call)
  File "/home/xxxx/.local/lib/python3.10/site-packages/maga_transformer/server/inference_server.py", line 166, in _infer_wrap
    rep = await self._infer_impl(req, id, raw_request, generate_call)
  File "/home/xxxx/.local/lib/python3.10/site-packages/maga_transformer/server/inference_server.py", line 264, in _infer_impl
    raise asyncio.CancelledError("client disconnects")
asyncio.exceptions.CancelledError: client disconnects

这个是有影响的吗？

这也是正常现象，是在等 gpu0 同步数据
client 强行断掉之后 _infer_impl raise asyncio.CancelledError("client disconnects") 是属于正常现象


[BUG] mx.random.normal generates -inf

Describe the bug
mx.random.normal behaves differently when setting seed or setting key. And it generates inf sometimes.
To Reproduce
Include code snippet

import mlx.core as mx

shape = (1, 64, 64, 4)
mx.random.seed(0)
x = mx.random.normal(shape=shape, dtype=mx.float16)
print(x.min(), x.max())

key = mx.random.key(0)
x = mx.random.normal(shape=shape, dtype=mx.float16, key=key)
print(x.min(), x.max())

outputs on my machine is
array(-inf, dtype=float16) array(3.29688, dtype=float16)
array(-3.48633, dtype=float16) array(3.29688, dtype=float16)

Expected behavior
The 2 prints should print same value. And inf should never be part of mx.random.normal, given its extremely low probability.
Desktop (please complete the following information):

OS Version: MacOS 14.1
Version: current main (e6418781)

The fact that seed and key are different is expected. Seeding is not equivalent to passing an explicit key. The equivalent is (but this is an implementation detail that shouldn't be relied on):
key = mx.random.key(0)
subkey, key = mx.random.split(key)
x = mx.random.normal(shape=shape, dtype=mx.float16, key=key)
print(x.min(), x.max())

As for the inf, that is a known issue with generating random numbers in fp16. For now I would suggest you use fp32 in the call to random.normal and then cast to the lower precision until we fix it.
I'm closing this as a duplicate of #642

The fact that seed and key are different is expected. Seeding is not equivalent to passing an explicit key. The equivalent is (but this is an implementation detail that shouldn't be relied on):
key = mx.random.key(0)
subkey, key = mx.random.split(key)
x = mx.random.normal(shape=shape, dtype=mx.float16, key=key)
print(x.min(), x.max())

👍 Is it suggested to always use key, so that things are stateless?

As for the inf, that is a known issue with generating random numbers in fp16. For now I would suggest you use fp32 in the call to random.normal and then cast to the lower precision until we fix it.
I'm closing this as a duplicate of #642

Thanks!

Is it suggested to always use key, so that things are stateless?

Not necessarily, no. There are cases where you have to go that route, for example when vmapping with random number generation inside a function. But in most cases actually the stateless requirement is not that strict and using the global RNG state is actually fine.


Problem with password field for creating WhatsApp instance

I'm new with WhatsApp API . As i understand i need to use some WART for creating unique password .
No version of WART works for me.
Also method WhatsAppApi.Register.WhatsRegisterV2.RequestCode doesn't work.
The same reason for all is OLD version.
In some place i saw that somebody wrote that WhatsApp closed the way the WhatsApp API works . Is it true? Are you still use it or it's obsolete project?
I asked these questions because i saw all of them in internet and there were not good answers for them...
Thanks in advance.

try this configuration :
using System;
using System.Globalization;
namespace WhatsAppApi.Settings
{
    /// <summary>
    /// Holds constant information used to connect to whatsapp server
    /// </summary>
    public class WhatsConstants
    {
        #region ServerConstants

        /// <summary>
        /// The whatsapp host
        /// </summary>
        public const string WhatsAppHost = "c3.whatsapp.net";

        /// <summary>
        /// The whatsapp XMPP realm
        /// </summary>
        public const string WhatsAppRealm = "s.whatsapp.net";

        /// <summary>
        /// The whatsapp server
        /// </summary>
        public const string WhatsAppServer = "s.whatsapp.net";

        /// <summary>
        /// The whatsapp group chat server
        /// </summary>
        public const string WhatsGroupChat = "g.us";

        /// <summary>
        /// The whatsapp version the client complies to
        /// </summary>
        //public const string WhatsAppVer = "2.13.21";
        //public const string WhatsAppVer = "2.12.440";
        //public const string WhatsAppVer = "2.12.556";
        public const string WhatsAppVer = "2.17.337";

        /// <summary>
        /// The port that needs to be connected to
        /// </summary>
        public const int WhatsPort = 443;

        /// <summary>
        /// iPhone device
        /// </summary>
        //	public const string Device = "S40";
        public const string Device = "Android";

        /// <summary>
        /// manufacturer
        /// </summary>
        public const string Manufacturer = "HTC";

        /// <summary>
        /// OS Version
        /// </summary>
        public const string OS_Version = "4.3";

        /// <summary>
        /// The useragent used for http requests
        /// </summary>
		//public const string UserAgent = "WhatsApp/2.13.21 S40Version/14.26 Device/Nokia302";
        //public const string UserAgent = "WhatsApp/2.12.440 Android/4.3 Device/Xiaomi-HM_1SW";
        public const string UserAgent = "WhatsApp/2.17.337 Android/4.3 Device/endeavoru-IMM76D";

        #endregion

        #region ParserConstants
        /// <summary>
        /// The number style used
        /// </summary>
        public static NumberStyles WhatsAppNumberStyle = (NumberStyles.AllowDecimalPoint | NumberStyles.AllowLeadingSign);

        /// <summary>
        /// Unix epoch DateTime
        /// </summary>
        public static DateTime UnixEpoch = new DateTime(0x7b2, 1, 1, 0, 0, 0, 0, DateTimeKind.Utc);
        #endregion
    }
}

@ahll This configuration is worked for you?

I have a fork , now is working with recieving message, but still has problem with sending message.


prediction from classifier

I have 2 questions.

To make a prediction with a trained classifier, the README instructions say you need to have a file named test.tsv in the input folder. What is the input folder and how do I specify it in the call to make a prediction below?

python run_classifier.py
--task_name=COLA
--do_predict=true
--data_dir=$GLUE_DIR/CoLA
--vocab_file=$BERT_BASE_DIR/vocab.txt
--bert_config_file=$BERT_BASE_DIR/bert_config.json
--init_checkpoint=$TRAINED_CLASSIFIER
--max_seq_length=128
--output_dir=/tmp/cola_output/

Also, what is the path to the trained classifier? Does the export look something like the following?

export TRAINED_CLASSIFIER = /tmp/cola_output/model.ckpt-1603
i.e. is the path to the trained classifier the path to the latest model checkpoint?

--data_dir=$GLUE_DIR/CoLA is the  input folder

thank you HaodaY

i have the same question to ask, there is not the specific path of TRAINED_CLASSIFIER in the README instructions, have u alreadly pointed out the question?

I figured out the answer to my question #2 above. The weights of the trained classifier will be in the output_dir directory you specified as a parameter when your trained the classifier.
So in my example above, the weights of the trained classifier were saved to
--output_dir=/tmp/cola_output/
Look in that folder to see what the highest numbered model.ckpt file is and then set TRAINED_CLASSIFIER to something like:
export TRAINED_CLASSIFIER = /tmp/cola_output/model.ckpt-1603


python311Packages.aiovodafone: 0.2.1 -> 0.3.1

Diff: https://github.com/chemelli74/aiovodafone/compare/refs/tags/v0.2.1...v0.3.1
Changelog: https://github.com/chemelli74/aiovodafone/blob/0.3.1/CHANGELOG.md
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 257591 run on x86_64-linux 1

5 packages built:

home-assistant-component-tests.vodafone_station
python310Packages.aiovodafone
python310Packages.aiovodafone.dist
python311Packages.aiovodafone
python311Packages.aiovodafone.dist


Add test for multiline strings with commas inside arrays

like regular strings, multiline strings with commas in them should not interfere with the parsing of arrays.
this test exercises an edge case in the python library toml: https://github.com/uiri/toml/issues/319
>>> toml.load(open('tests/valid/array/string-with-comma-2.toml'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.10/site-packages/toml/decoder.py", line 156, in load
    return loads(f.read(), _dict, decoder)
  File "/usr/lib/python3.10/site-packages/toml/decoder.py", line 511, in loads
    ret = decoder.load_line(line, currentlevel, multikey,
  File "/usr/lib/python3.10/site-packages/toml/decoder.py", line 767, in load_line
    k, koffset = self._load_line_multiline_str(pair[1])
  File "/usr/lib/python3.10/site-packages/toml/decoder.py", line 797, in _load_line_multiline_str
    while len(newp) > 1 and newp[-1][0] != '"' and newp[-1][0] != "'":
IndexError: string index out of range

Thanks


manifest-template.xml.envsubst: use new repo location instead of forwards

Maybe we can also remove the docker and systemd repos here since they are not used anymore

Maybe we can also remove the docker and systemd repos here since they are not used anymore

Need to check if LTS also stopped using them

The systemd repo can be dropped, but LTS has the ebuilds for rkt and docker (1.12, 17) pointing to the repos


php5 -> php7

Great work! :-)
With actual raspbian stretch this line from installation manual doesn't work anymore:
sudo apt-get install php5-common php5-cgi php5
PHP5 isn't available anymore as installation package.
It should be substituted simply with:
sudo apt-get install php-common php-cgi php
This installs PHP7 and everything works as it should.

Can you use the playbutton for each dictornery on the Webinterface?

The same here. I can't use them.
Never tried it before.

I looked into it. Maybe it's only a typo.
In this button there is in htdocs/index.php written /usr/bin/cvlc instead of /usr/bin/vlc:
I will try it tomorrow - if this repair this issue.
// play folder with VLC if(isset($urlparams['play']) && $urlparams['play'] != "" && is_dir($urlparams['play'])) {     // kill vlc if running     // NOTE: this is being done as sudo, because the webserver does not have the rights to kill VLC     $exec = "/usr/bin/sudo pkill vlc > /dev/null 2>/dev/null &";     exec($exec);     // pipe playlist into VLC     // NOTE: this is being done as sudo, because the webserver does not have the rights to start VLC     $exec = "/usr/bin/sudo /usr/bin/cvlc -I rc --rc-host localhost:4212 ".$urlparams['play']." > /dev/null 2>/dev/null &";     exec($exec);     /* redirect to drop all the url parameters */     header("Location: ".$conf['url_abs']);     exit;  }

cvlc is the original command line interface for vlc. Not sure if the c is still needed today. I am planning to add the --no-video option to the script, just to be sure this will only play sound.
https://wiki.videolan.org/Video_Output/

Yes, you are right of course.
I changed the 2>/dev/null part of the cvlc play command to a logfile and got the following output:
[01355ef0] pulse audio output error: PulseAudio server connection failure: Connection refused
[0136eb40] core interface error: no suitable interface module
[012bd0f0] core libvlc error: interface "globalhotkeys,none" initialization failed
[0136eb40] dbus interface error: Failed to connect to the D-Bus session daemon: Unable to autolaunch a dbus-daemon without a $DISPLAY for X11
[0136eb40] core interface error: no suitable interface module
[012bd0f0] core libvlc error: interface "dbus,none" initialization failed
libdvdnav: Using dvdnav version 5.0.3
[0136eb40] [cli] lua interface: Listening on host "localhost:4212".
libdvdread: Encrypted DVD support unavailable.
************************************************
**                                            **
**  No css library available. See             **
**  /usr/share/doc/libdvdread4/README.css     **
**  for more information.                     **
**                                            **
************************************************
libdvdread: Couldn't find device name.
libdvdread:DVDOpenFilePath:findDVDFile /VIDEO_TS/VIDEO_TS.IFO failed
libdvdread:DVDOpenFilePath:findDVDFile /VIDEO_TS/VIDEO_TS.BUP failed
libdvdread: Can't open file VIDEO_TS.IFO.
libdvdnav: vm: failed to read VIDEO_TS.IFO
[0136eb40] [cli] lua interface error: Error loading script /usr/lib/arm-linux-gnueabihf/vlc/lua/intf/cli.luac: lua/intf/modules/host.lua:279: Interrupted.

I don't understand where the "piping the playlist to vlc" takes place?
The following command manually started as user pi works (without --no-video and with):
´´´
/usr/bin/sudo /usr/bin/cvlc  -I rc --rc-host localhost:4212 file:///home/pi/RPi-Jukebox-RFID/playlists/Disney_1.m3u
´´´

It will be same command, if we use a RFID card.
I am not a programme but my thinking is:
kill cvlc
play folder which is mark

Sorry, I made yesterday a mistake in testing your suggestion with the "--no-video" option.
I checked it again today and now everything works (with this additional option and not without)!
:-)
It seems that this option is needed. Now the function in index.php looks like this:
// play folder with VLC
if(isset($urlparams['play']) && $urlparams['play'] != "" && is_dir($urlparams['play'])) {
    // kill vlc if running
    // NOTE: this is being done as sudo, because the webserver does not have the rights to kill VLC
    $exec = "/usr/bin/sudo pkill vlc > /dev/null 2>/dev/null &";
    exec($exec);

    // pipe playlist into VLC
    // NOTE: this is being done as sudo, because the webserver does not have the rights to start VLC
    $exec = "/usr/bin/sudo /usr/bin/cvlc --no-video -I rc --rc-host localhost:4212 ".$urlparams['play']." > /dev/null 2>/dev/null &";
    //$exec = "/usr/bin/sudo /usr/bin/cvlc --no-video -I rc --rc-host localhost:4212 ".$urlparams['play']." > /home/pi/vlc1.txt 2>/home/pi/vlc2.txt &";
    exec($exec);
    /* redirect to drop all the url parameters */
    header("Location: ".$conf['url_abs']);
    exit;
}


Profile tip button is showing even if user hasn't set up a Lightning Address

Probably shouldn't show the lightning tip button on a user's profile if they don't have a Lightning Address set on their profile.
https://snort.social/p/npub1nw5vdz8sj89y3h3tp7dunx8rhsm2qzfpf8ujq9m8mfvjsjth0uwqs9n2gn
Yellow's profile on
Snort

vs
Damus

good catch, fixed in https://github.com/v0l/snort/pull/199


Size mismatch error when loading weights from pretrained weights (hrnetv2-c1)

I am training on my custom dataset using transfer learning but getting the size mismatch error because of the different number of classes on my dataset (binary) and the weights (trained on 150 classes). Do I need to unfreeze last layers or is there some other hack to get this to work. Thanks!
[2020-06-08 04:52:04,191 INFO train.py line 248 4485] Running with config:
DATASET:
  imgMaxSize: 1000
  imgSizes: (300, 375, 450, 525, 600)
  list_train: ./data/stock_training.odgt
  list_val: ./data/stock_validation.odgt
  num_class: 2
  padding_constant: 32
  random_flip: True
  root_dataset: .
  segm_downsampling_rate: 4
DIR: ckpt/ade20k-hrnetv2-c1
MODEL:
  arch_decoder: c1
  arch_encoder: hrnetv2
  fc_dim: 720
  weights_decoder:
  weights_encoder:
TEST:
  batch_size: 1
  checkpoint: epoch_9.pth
  result: ./inference_output
TRAIN:
  batch_size_per_gpu: 2
  beta1: 0.9
  deep_sup_scale: 0.4
  disp_iter: 20
  epoch_iters: 5000
  fix_bn: True
  lr_decoder: 1e-06
  lr_encoder: 1e-06
  lr_pow: 0.9
  num_epoch: 35
  optim: SGD
  seed: 304
  start_epoch: 30
  weight_decay: 0.0001
  workers: 16
VAL:
  batch_size: 1
  checkpoint: epoch_30.pth
  visualize: False
[2020-06-08 04:52:04,191 INFO train.py line 253 4485] Outputing checkpoints to: ckpt/ade20k-hrnetv2-c1
Using weights: ckpt/ade20k-hrnetv2-c1/encoder_epoch_30.pth ckpt/ade20k-hrnetv2-c1/decoder_epoch_30.pth
Loading weights for net_encoder
Loading weights for net_decoder
Traceback (most recent call last):
  File "train.py", line 282, in <module>
    main(cfg, gpus)
  File "train.py", line 158, in main
    weights=cfg.MODEL.weights_decoder)
  File "/home/ubuntu/autonopia/semantic-segmentation-pytorch/models/models.py", line 166, in build_decoder
    torch.load(weights, map_location=lambda storage, loc: storage), strict=False)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 847, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for C1:
        size mismatch for conv_last.weight: copying a param with shape torch.Size([150, 180, 1, 1]) from checkpoint, the shape in current model is torch.Size([2, 180, 1, 1]).
        size mismatch for conv_last.bias: copying a param with shape torch.Size([150]) from checkpoint, the shape in current model is torch.Size([2]).```

Just had to keep the num_class = 2 in the config and it continued training.


ccache: 4.7.3 -> 4.7.4

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for ccache is: "Compiler cache for fast recompilation of C/C++ code"
meta.homepage for ccache is: "https://ccache.dev"
meta.changelog for ccache is: ""
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 4.7.4 with grep in /nix/store/2fz6dj6rsn7nf4948brazdrscxwm5mrp-ccache-4.7.4
found 4.7.4 in filename of file in /nix/store/2fz6dj6rsn7nf4948brazdrscxwm5mrp-ccache-4.7.4

Rebuild report (if merged into master) (click to expand)

3 total rebuild path(s)

3 package rebuild(s)

First fifty rebuilds by attrpath
ccache
ccacheStdenv
ccacheWrapper

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/2fz6dj6rsn7nf4948brazdrscxwm5mrp-ccache-4.7.4 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A ccache https://github.com/r-ryantm/nixpkgs/archive/4df4d16a87f8826b95a8259730245501464c4e6c.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/2fz6dj6rsn7nf4948brazdrscxwm5mrp-ccache-4.7.4
ls -la /nix/store/2fz6dj6rsn7nf4948brazdrscxwm5mrp-ccache-4.7.4/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

3 packages built:

ccache
ccacheStdenv
ccacheWrapper

Maintainer pings
cc @kira-bruneau @r-burns for testing.

Result of nixpkgs-review pr 202570 run on x86_64-linux 1

3 packages built:

ccache
ccacheStdenv
ccacheWrapper


Slider control application examples update

Overview
Added examples of using Slider control's value.
Rotate object

Scale object

Change color

Changes

Updated SliderExample scene
Added example scripts that use Slider control's value (SliderRotateObject.cs, SliderChangeColor.cs, SliderScaleObject.cs)

Requires
This PR requires @julenka 's PR #4022. It should be merged first. Separated PR for easier review.

Hi Yoon, the sliders PR #4022 is in! I changed the public fields of the start and end points of the sliders, as well as updated some of the Animation state machines, so you will probably need to re-import the prefabs. Please see the pinchslider example scene to see examples of the new sliders.
Thank you again for making this great example!

Resolved scene merge conflict. Thanks!

This looks fantastic.
It might be worth changing the thumb sliders on one example to show how easy it is. I swapped in platonic meshes (which are much larger than the slider thumb mesh) and the scale / color animations still worked perfectly after I scaled the thumb transform.

Nitpick - perhaps we should disable the hand mesh / hand joint buttons on platforms where these have no effect?

/azp run mrtk_docs

/azp run mrtk_ci

Polished the value label typography.
Removed Rotate/Scale example per Nick's feedback - we recommend direct manipulation for the rotate/scale, based on our learnings from previous projects.

Yoon, do you still have the rotate and scale examples in the scenes? I thought we wanted those removed.

/azp run mrtk_ci

/azp run mrtk_docs

/azp run mrtk_ci


[grpc] Fix gRPC++ reflection library is missing for x86-windows triplet

Fixes https://github.com/microsoft/vcpkg/issues/30587
When installing under x86-windows, add a prompt message to complete the installation of the required lib libraries.

[X] Changes comply with the maintainer guide
[ ] SHA512s are updated for each updated download
[ ] The "supports" clause reflects platforms that may be fixed by this new version
[ ] Any fixed CI baseline entries are removed from that file.
[ ] Any patches that are no longer applied are deleted from the port's directory.
[X] The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
[X] Only one version is added to each modified port's versions file.

All features passed with following triplets:
x86-windows
x64-windows

I think the user just needs to be told they need the codegen feature. The only reason they are getting those bits without it is
    {
      "name": "grpc",
      "host": true,
      "features": [
        "codegen"
      ]
    }

in the dependencies list and that x64-windows is the host architecture.

Okay, I will reply to the user what our 'host' means and then close this PR.


Add support for extensions

https://www.prisma.io/docs/concepts/components/prisma-schema/postgresql-extensions
With the current implementation, this code:
datasource: {
  // rest
  extensions: `[pg_tgrm, zombodb]`,
  // rest
}

Evaluates to:
extensions = "[pg_tgrm, zombodb]"

@KrzysztofZawisla What if you provide an array of strings? Can add type support as well, but that should work unless I'm horribly wrong.

@KrzysztofZawisla Added to https://github.com/ridafkih/schemix/pull/53, if you'd like to test, otherwise I'll have this merged in tomorrow! :)

Has been merged and released in v1.8.4

@KrzysztofZawisla What if you provide an array of strings? Can add type support as well, but that should work in the meantime unless I'm horribly wrong.

Sorry, I was out of programming vibe for these 4 days, so I didn't even check replies on this issue. It's important to not have items in array as string, because they will also be evaluated to string in prisma schema. Prisma requires passing extensions not as string to properly highlight syntax. In free time I will check the new release if it solves that problem.

@KrzysztofZawisla Thanks for letting me know! I'll make this adjustment tonight.

@KrzysztofZawisla Alright, #54 should address this fair and square. I was actually looking at the project you're contributing to the other day that implemented Schemix, and was impressed by the scale at which you guys took the library. Was very happy to see that.
Since you guys are really using the library well, if you have any other suggestions, feature requests, or bug reports, let me know and I'll be sure to address them as soon as I can.

Released in v1.8.5 since the change does not touch any other components, let me know if that addresses that and I will close this issue.

@ridafkih I have checked the new release (1.8.5) and everything works perfectly good.
I'm also happy about the fact that you have created that library, and we found it. Unique and well-done documented library in this field. Our Prisma schema expanded dramatically, and we thought that have to be separated + we had issues with stuff that must have done programmatically like country codes or enums generated based on typescript enums. We will make a PR or report issue in case when we will found a way to improvement something, but firstly we need to make our PostgreSQL, MongoDB, and FaunaDB schemas/structure and migrations 😅.
Best Regards,
Krzysztof Zawisła and @borys-malinowski


Added testcase for creating pod with private registry image (test_k8s…

…_env_create_pod_with_private_registry_image)
@sangeethah : Please take a look.

@soumyalj , we should not be executing this test case , when parameters relating to private registry is not passed. Make sure that these test cases get skipped when private registry parameters are not available.

Also cab u include include the yml  file needed for this test?

@sangeethah : Handled test case skip and also checked in the yml file

LGTM


Update DG logic image and remove undo/redo section

Key Summary

Update ParserClasses.png
remove undo/redo section of the DG

Codecov Report

Merging #217 (0dea691) into master (d918f86) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #217   +/-   ##
=========================================
  Coverage     67.71%   67.71%
  Complexity      550      550
=========================================
  Files            96       96
  Lines          1908     1908
  Branches        245      245
=========================================
  Hits           1292     1292
  Misses          531      531
  Partials         85       85

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d918f86...0dea691. Read the comment docs.


chore: transaction evm input mismatch should not be an error

PR Type
Enhancement

Description

Enhanced error handling in the save_execution method of StratusStorage implementation
Added specific handling for TransactionEvmInputMismatch error:

Now logs a warning instead of an error
Indicates that the operation will be retried

Maintained error logging for other types of errors
Improved overall robustness of the transaction execution saving process

Changes walkthrough 📝
Relevant filesError handling

stratus_storage.rsImprove error handling for transaction execution saving

src/eth/storage/stratus_storage.rs
Modified error handling in save_execution method  Added specific handling for TransactionEvmInputMismatch error  Changed logging level from error to warn for input mismatch cases

+6/-2

💡 PR-Agent usage: Comment /help "your question" on any pull request to receive relevant information

Final benchmark:
Run ID: bench-2366532462
Git Info:

Branch: main
Commit: f9ecb62
PR: #1891

Configuration:

Target Account Strategy: Default (Random)

RPS Stats: Max: 1185.00, Min: 756.00, Avg: 1016.40, StdDev: 68.25
TPS Stats: Max: 1164.00, Min: 861.00, Avg: 985.20, StdDev: 72.86
Plot: View Plot


Swiper shows slide 2 unexpetedly

Check that this is really a bug

[X] I confirm

Reproduction link
https://codesandbox.io/s/swiper-infinite-loop-angular-forked-y85mt?file=/src/app/app.component.ts
Bug description
Under certain circumstances, slide no2 will be shown instead of the correct slide. This seems to be connected to the (now deprecated) [(index)] attribute.
This but appears with the change from v7.0.5 -> v7.0.6
This looks like the offending line. I'm happy to do a PR but am not sure if the line can just be removed or if there's a greater context I'm missing
Expected Behavior
The slide at the given index should be displayed
Actual Behavior
Slide no2 is displayed
Swiper version
7.0.6
Platform/Target and Browser Versions
tested on iPadOS Safari 15 & Chrome Version 93.0.4577.82 (Official Build) (64-bit)
Validations

[X] Follow our Code of Conduct
[X] Read the docs.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.
[X] Make sure this is a Swiper issue and not a framework-specific issue

Would you like to open a PR for this bug?

[X] I'm willing to open a PR

Yeah, seems like that line came from local testing. PR to remove it is welcome 🙂


I Just Lost My Manual Fan Curve :-(

I had created very particular fan curves. I was on GPU Internal Thermistor and clicked the checkbox for Fan 3, and the fan curve grid went blank. When I clicked Fan 3 checkbox again, my custom fan curve was gone. Not good!! You absolutely must keep the fan curve somewhere, permanently unless manually deleted, and restore it when the appropriate checkbox is reeneabled.

Work as designed right now, unchecking fan remove curve.
In the next release, i change this behavior, as well as many others.


Unexpected behaviour of Volume.legosurface()?

Hello,
I've noticed a strange behaviour of Volume.legosurface() method trying to visualize 3D binary matrix (boolean values representing object/background) and/or 3D label matrix (integer values representing different regions).
To illustrate this, consider the following example:
import numpy as np

data_matrix = np.zeros([10, 10, 10], dtype=np.uint8)
# all voxels have value zero except:
data_matrix[0:2, 0:2,  0:2] = 1
data_matrix[4:5, 4:5, 4:5] = 2
data_matrix[7:10, 7:10, 7:10] = 3

from vtkplotter import Volume, Plotter

vol = Volume(data_matrix)

EPS = 1e-6
msh = vol.legosurface(vmin=1-EPS, vmax=3+EPS, cmap=['red', 'green', 'blue'])

vp = Plotter(bg="black")
vp.add(msh)

vp.show(axes=1)

Issue no. 1:
I think that parameters vmin and vmax are not properly described in the documentation, i.e. the documentation states that

vmin (float) – the lower threshold, voxels below this value are not shown.
vmax (float) – the upper threshold, voxels above this value are not shown.
but it seems that parameters vmin and vmax define a range (i.e. (vmin, vmax)) not the interval (i.e. [vmin, vmax]) -- not a big deal (can be handled with EPS or something similar) but can cause unexpected behaviour.

Issue no. 2:
My expectations of legosurface() behaviour was that the code listed above will plot three cubes: 1x1x1 (red), 2x2x2 (green) and 3x3x3 (blue). In other words, I expected to see "voxelized" plot of these three regions. However, vtkplotter produced 0x0x0 (red), 1x1x1 (green) and 2x2x2 (blue) regions: https://prnt.sc/rmk0yo
Whether this is a bug or I didn't understand the usage of this method well?
It is not a big problem for many use-cases, but I am trying to use vtkplotter to visualize image segmentation output in 3D, where each voxel counts. :-)
Thank you!

Hi @kemo993
thanks for drawing my attention to these issues, give me one day to look more carefully at them!
and get back to you asap, M

issue 2 seems quite tricky because vtk seems to associate the scalar not to the center of the voxel but to the edge so effectively reducing the nr of voxels by 1... i'll see if i can think of a way around this.. one possibility would be to use a Glyph object.
also, I suspect you are using an old version.. try
pip install vtkplotter -U
you should be then able to reproduce this
import numpy as np
data_matrix = np.zeros([10, 10, 10], dtype=np.uint8)
data_matrix[0:3, 0:3,  0:3] = 1
data_matrix[3:7, 3:7,  3:7] = 2
data_matrix[7:10,7:10,7:10] = 3
#data_matrix = np.insert(data_matrix, 0, data_matrix[0,:,:], axis=0)
#data_matrix = np.insert(data_matrix, 0, data_matrix[:,0,:], axis=1)
#data_matrix = np.insert(data_matrix, 0, data_matrix[:,:,0], axis=2)
#print(data_matrix)

from vtkplotter import Volume, show

vol = Volume(data_matrix).c(['white','r','g','b'])

msh = vol.legosurface(vmin=1, vmax=3, cmap=['red', 'green', 'blue'])

iso = vol.isosurface(threshold=0.99).lw(0.1)

show(vol, msh, iso, N=3, axes=1)

note that the nr of cubes in each axis is 9, not 10.

Hi @marcomusy
I understand, this is how vtk works with legosurface.
Thank you, your suggestion to look into Glyph object really helped me a lot and I think I figured out how to get the visualization similar to legosurface without reducing the number of voxels.  If anyone needs it, here is a code snippet and corresponding output:
import numpy as np

data_matrix = np.zeros([10, 10, 10], dtype=np.uint8)
# all voxels have value zero except:
data_matrix[0:2, 0:2,  0:2] = 1
data_matrix[4:5, 4:5, 4:5] = 2
data_matrix[7:10, 7:10, 7:10] = 3

from vtkplotter import Volume, Plotter, shapes

vol = Volume(data_matrix)

EPS = 1e-6
msh = vol.legosurface(vmin=1-EPS, vmax=data_matrix.max()+EPS)

gly_cube = shapes.Cube(pos=(0, 0, 0), side=0.90)
orvs = np.array([data_matrix[tuple(v.astype(int))] * np.array([1., 0., 0.]) for v in msh.points()])
gly_msh = shapes.Glyph(msh, gly_cube, c='jet', orientationArray=orvs)

vp = Plotter(bg="black")
vp.add(gly_msh)

vp.show(axes=1)

vp.close()

Probably not the most elegant way, but I exploited orientationArray to pass corresponding values to colormap (slightly different from the problem I explained above, but 'jet' colormap is exactly what I need).

Nice! I had a look at it yesterday and came up with a similar solution.. I'll see if can improve on the Glyph class to make it a bit more versatile..

In the lastest version
pip install -U vtkplotter
you can also do something like:
from vtkplotter import *
import numpy as np

data_matrix = np.zeros([10, 10, 10], dtype=np.uint8)
# all voxels have value zero except:
data_matrix[0:2, 0:2, 0:2 ] = 1
data_matrix[4:5, 4:5, 4:5 ] = 2
data_matrix[7:10,7:10,7:10] = 3

scals = data_matrix.ravel(order='F').astype(np.float)
zeromat = np.zeros(data_matrix.shape+np.array([1,1,1]), dtype=np.uint8)
zvol = Volume(zeromat)

pts = Points(zvol.cellCenters()+.5).addPointScalars(scals,'glyphs')
obj = Cube(side=1/np.max(data_matrix))
gl = Glyph(pts, obj, c='jet', scaleByScalar=True).lw(0.1)

show(gl, axes=1)

Thank you, this looks much more elegant compared to the trick with dummy orientation vectors scaled by scalar values.


[Spike] Figure out strategy around backwards compatibility and support for older OCP versions

We seem to be hitting walls regarding when we can upgrade to catch up to upstream, example from v1.7 upgrade thread:

1.7 Requires Tekton 0.47. This is introduced in OCP Pipeline 1.11. The earliest OCP adoption for this is 4.12. Which means We have to wait until OCP EOL's 4.11 before proceeding with this change.

We constantly face obstacles around having to support older OCP due older OCP Pipelines versions when looking to upgrade dsp (kfp). We should formalize what the process/policy ought to be here, and push for a practical solution whereby we are not constantly blocked on releases due to EOL support dates for OCP.
Some questions to answer:

Is it a hard requirement we wait until an OCP version reaches EOL before upgrading to a specific OCP Pipelines version?
Can we support older/newer versions in a given release? (olm / dspa / odh-dashboard feature enablement options)

Migrated to Jira: https://issues.redhat.com/browse/RHOAIENG-1664


Syntax highlighting in C++ headers

I noticed that Atom highlights .h files as C although they can be C++ as well. So keywords like class or namespace are not highlighted.
I think it would be nice to have some sort of automatic C vs. C++ detection. Or maybe you could just set it to C++ by default, they both have the same keywords anyway.

A workaround is to add this to your ~/.atom/init.coffee:
# Remove ".h" from C grammar (use C++ instead):
atom.grammars.onDidAddGrammar (grammar) ->
  if grammar.name is "C"
    grammar.fileTypes.splice(grammar.fileTypes.indexOf("h"), 1)

@randomstuff thanks, it works, but there's a ) missing at the end of indexOf("h"), 1

We plan to make the mapping of file extensions to grammars configurable sometime soon.

We plan to make the mapping of file extensions to grammars configurable sometime soon.

:sparkling_heart:

@andreldm, Whoops copy/paste error. Thanks.

I tried the file-types package which makes it easily possible to overwrite file type association in the ~/.atom/config-cson. I just added this line:
  "file-types":
    h: "source.cpp"

(https://atom.io/packages/file-types)

@patgithub That functionality is built into Atom. No need to use a package, just use the core.customFileTypes option. This was only recently documented however, and the documentation doesn't seem to have gone live just yet, so for the time being check out:
https://github.com/atom/docs/blob/6e8baeecaf939426e66ee960c168cf27117215ef/book/02-using-atom/sections/06-customizing.asc#customizing-language-recognition

@thomasjo much thanks - works good too!

@patgithub Really works .thx for your help

Thanks for all the foregoing suggestions!
I tried the easy way selecting manually C++ with the grammar-selector package.
I don't get any error now, but still I have couple of red points and that's the error message I get:

even though I get the green light and no issues at the bottom.
How is such a behaviour possible?

@inspiralpatterns That's an issue with whatever linter package you're using.


Only propagate transforms entities with GlobalTransforms.

Objective
Fixes a performance issue when you have 1000s of entities in a bevy hierarchy without transforms.
This was prominently happening in bevy_ecs_tilemap.
Solution
Filter out entities that don't have a global transform.
Testing
CI
We should test some other way...
Migration Guide

To avoid surprising performance pitfalls, Transform / GlobalTransform propagation is no longer performed down through hierarchies where intermediate parent are missing a GlobalTransform. To restore the previous behavior, add GlobalTransform::default to intermediate entities.

Just to be clear in a scenario where you have this tree:
Transform entity> no transform entity > transform entity propagation would not flow up the tree.

Approved, but I want to point out that this might be a surprising change for people who use organizational entities in their tree with only names, e.g.

Player (has transform)

PhysicsStuff (has no transform)

Collider (has transform)
Joint (has transform)
Sensor (has transform)

This code would now suddenly break in a subtle way. Maybe a migration guide would be good or some documentation would be good.

I wrote up a nice little migration guide, and added a bit more context to the PR description :)

I think that it's showing something very wrong is happening.
Transform/GlobalTransform are supposed to be on all levels of the hierarchy. If you have holes, what is happening is pretty much undefined, and you're not using transforms in the way expected by the engine.
This should at minimum be a warning, at best be rejected

This was prominently happening in bevy_ecs_tilemap.

What is bevy_ecs_tilemap doing to have that, and what does it expect?

I assume they don't have holes but just big hierarchies without any transforms and the PR just wants to note that holes are no longer supported as a consequence

I assume they don't have holes but just big hierarchies without any transforms, which should be supported. I think the note about holes no longer being supported is just pointing out the logical consequences of this PR.
I would be careful with adding checks for warnings to transform propagation as it's quite a hot path. If they're added, I'd like to see a benchmark.

Bevy already has validation for components that should exist uninterrupted all the way up the tree, but they run in a separate system. There is no reason validation logic has to live inside the propagation code.


[Bug] Evaluation requirements not clear enough

Describe the bug
Evaluation requirements can be misunderstood, specifically where "service" and "stage" flags are coming from. A link to corresponding workflow (e.g. getting started) needs to be made more clear.

updated documentation with examples for each pipeline to clarify how to configure this, also video tutorials are being created to futher document the process


hello again,

hey we get a 1 detector on virustotal if we do the build and if we do with self icon we get 10 detectors on virustotal i hope it will get fixed and the roblox cookie dosent shows on webhook only in cookies.txt and yeah

Paid for FUD is not free 😂


Implement support to Sanitize on v3

As listed at #715 the Sanitize actions are not yet working on v3.
https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#sanitiseArg

Typo fixed at: c2bc6952651707f773280b7b8188db814aa11718

Hi, is there an update on this feature, when can we expect it? Is it currently planned for v3.1.1?

Hello @Veszner ,
The current plan is to implement some sanitization support in v3 a little later this year -- not in the currently-active v3.0.7, but in the v3 work immediately thereafter.

Any news on this issue ?

Hello @todd-richmond ,
Regarding "Can't upgrade from v2 w/o it" ...
The lack of sanitization capability is considered a gap in current v3 functionality -- and one that is desirable to fill.
But there is no particular reason to view migrating from ModSecurity v2 to ModSecurity v3 as important or highly desirable.
ModSecurity v2 continues to be the maintained version of the software that is suitable for use with Apache HTTP Server (and IIS), while ModSecurity v3 is the maintained version that is suitable for use with nginx.

Any news on that matter ?
Password (non)removal from logs is a serious security concern, especially for a security product.

is there any plan for this to be solved?


fix rst format

Changes

fix #334  warnings

TODO

[x] Fix groff format

@junghans  these changes should have fixed the rst warnings

@junghans  these changes should have fixed the rst warnings


Add an information need to simulator

Define an information need for the simulator. This information need will be used to filled elicited slots and make a decision regarding a recommended item.

Next steps are:

Add a property information_need to UserSimulator
Update agenda-based user simulator to take into consideration information need when generating its replies


Fast rollout features

Need to implement features described in Extended Data Table 4

[ ]  Response
[ ]  Save atari
[ ] Neighbor
[ ] Nakade
[ ] Response pattern
[ ] Non-response pattern
[ ] Last move distance

Note that 'self-atari' in this network can be the 'self atari' feature for the deep policy network with maximum=1

Did some thinking on this last night.  For the 'last move distance', my understanding it is the Manhattan Distance from the last two moves.  It supposedly gives 34 possible patterns.   Is it the sum of distances?  ie MHDist(p1, p2) + MHDist(p2, p3) or MHDist(p1, p3)+MHDist(p2, p3) or a tuple?  Or something else?  assuming it is the sum of distances - for the first dist method I get 53 possibilities and for the second I get 70 possibilities, both of which are quite a bit more than 34.  So need some clarification there.
For the patterns - are there pattern databases that we should be looking at?  I don't think the patterns can be copyrighted (they are 'merely functional')

It looks like they might be using the LGPLed libraries from Fuego, since the features it uses for decisions are basically the same list, here is the relevant code.
https://sourceforge.net/p/fuego/code/HEAD/tree/trunk/features/FeBasicFeatures.cpp
https://users.soe.ucsc.edu/~glin/docs/Fuego_Fall09Report.pdf

Here is the relevant code I've found so far...
Save Atari GoBoardUtil.h
Nakade	GoEyeUtil.cpp, .h
Non-response pattern -> GoPattern3x3.cpp
Response pattern -> GoPattern12Point.cpp
Self-atari -> GoBoardUtil.h
Last move distance -> FeBasicFeatures.cpp
Non-respnse pattern -> GoPattern12Point.cpp

So a question I have, that isn't clear from the paper - for the patterns - is this a massive vector that each pattern is a one hot?  or is it a single variable with a 1 if any of the patterns are present.

Thanks for all the resources! Regarding your last question, we interpreted it as a massive 1-hot encoding. Notice that the 'response' feature says "Whether move matches one or more response pattern features". So the features with >1 pattern must mean all of the possible patterns.. Actually building the full one-hot encoding for a 69338-dimensional vector and multiplying by a weight vector could be replaced by a quick lookup of the ith softmax weight if it turns out to be slow.

I emailed Aja for clarification, will let you know what they say.  I figured it is probably one hot encoding as well but doesn't hurt to ask, since I already needed clarification on their manhattan distance algorithm.

This is the function I have so far for the modified manhattan distance using information from Fuego.  Note that addition of max(dx, dy) and truncation which makes it not a true manhattan distance.
def truncatedManhattanDistance(p1, p2):
dx = abs(p1.x - p2.x)
dy = abs(p1.y - p2.y)
d = min(dx + dy + max(dx, dy), 17)
return d
def summedTruncatedManhattanDistancs(p1, p2, p3):
return truncatedManhattanDistance(p1, p3)+
truncatedManhattanDistance(p2, p3)

Looks like the 3x3 pattern code (and some other stuff as well) will be easier to adapt from here
https://github.com/pasky/michi/blob/master/michi.py
and for speed here is a c reimplementation of the same...
https://github.com/db3108/michi-c
Both are MIT licensed, which is good.

Ah the distance I thought they were using for Manhattan distance, I misunderstood it.  It is actually 'for gridcular' coordinates for the diamond pattern.
https://groups.google.com/forum/#!topic/computer-go-archive/2pQtpqyxSq8

Making good progress, will probably have a preliminary commit ready in the next day or so that has all or almost all of the features ready (though I'm not getting the same pattern counts...)

I tried enumerating all possible 3x3 patterns where the middle space is empty, taking into account the number of liberties (a detail in the paper methods), pruning out all symmetries, etc.. and came up with a number around 45000 total. Yet in the paper they claim to use 69338 3x3 patterns. Either they use some duplicate/impossible patterns, or I have a bug in my code. The bug seems likely at this point..

My sanity-check counts were way off... I just re-did them and found 486578 possible 3x3 patterns where

stone color and liberties (1,2,>=3) jointly make a position unique (from the paper's methods)
the middle is empty
all symmetric positions are removed
connected stones have the same (1,2, or >=3) liberties
each group has at least as many liberties as adjacent empty spaces visible in the 3x3 area
I ignored patterns with the edge of the board, which would only increase the number

So 69338 possible variations within the patterns used in Fuego may be more reasonable than I had thought. How did you check these pattern counts?

I'm currently only matching the patterns that are used in other Go systems.  See the michi code, which I've based it on...
https://github.com/pasky/michi/blob/master/michi.py
plus see my pull request (this filters out the impossible patterns)
https://github.com/pasky/michi/pull/11
Then I'll add in the liberty counts as well coloration information...
Really I think we should reduce the pattern count dramatically since the variants collapse to 14 or so common situations (double it for the color of the pattern...) then a multiplier again for number of liberties.

@LetterRip are you still working on this? If not, could you make a PR with the features you implemented?

@wrongu sorry for the delayed response.  No I'm not working on this.  I kept running into the problem that my features would be orders of magnitude different from those in their paper.  So I suspect I simply don't understand go well enough to do the implementations.

AG uses two fast policies: fast rollout policy and tree policy.
-- rollout policy

[ ] Response:   1 -> match response pattern
[ ] Save Atari: 1 -> move saves stones (i assume no ladder check?)
[ ] Neighbor:   8 -> 8-connected
[ ] Nakade:     8192 -> match nakade pattern at captured stone (why captured only?)
[ ] Response pattern: 32207 -> 12-point diamond pattern near previous move
[ ] Non-response pattern: 69338 -> 3x3 pattern around move
-- tree policy
[ ] Self-atari
[ ] Last move distance   -> manhattan distance with max 34
[ ] Non-response pattern -> 12-point diamond pattern around move

Explanation Extended Data Table 4 in paper:
"Features used by the rollout policy (first set) and tree policy (first and second set). Patterns are based on stone colour (black/white/empty) and liberties (1, 2, ≥3) at each intersection of the pattern."
AG only uses tree policy to create a placeholder next move prediction for a new MTCS tree node until a SL policy  analysis is preformed. Right now there is no tree policy included in MTCS framework so i only focused on the rollout policy features.
I gathered 3x3 and 12-point diamond shape patterns from ~51000 pro sgf files, results can be found here.
To me the most promising seems removing all rotational and colour symmetries taking the top ~4500 patterns (patterns used >~90) restoring all rotational and colour symmetries we have a ~69000 pattern count but only need a ~4500 one-hot vector
same goes for the 12D patterns  removing all rotational and colour symmetries taking the top ~2000 patterns (patterns used >~150) restoring all rotational and colour symmetries we have a ~32000pattern count but only need a ~2000one-hot vector

as discussed before Rémi Coulom suggest just using the patterns most frequently used has its drawbacks and suggests to calculate an elo ranking for each pattern on how successful it is.

an idea i was wondering about is to train a small network using all patterns and use the ones with the highest bias.

next to that i had a look at the nakade patterns but i am in doubt about their description: match nakade pattern at captured stone what do they mean with  "at captured stone"? it seems unlikely to me but are they only used after a stone capture?
i created a small collection of dead shapes (in nakade folder) and try to come up with an useful way to create a hash for them but so far i have not found a good way to do so. suggestions are welcome!
@wrongu & @prescod what kind of lookup would be most useful/fastest? also what kind of hash would you suggest?
@oriskunk you are working on using pachi rollout patterns, do you have to gather patterns to use with it? i only had a look at fuego and gnugo.

It's been a while since I've looked at this. What's the count of 3x3 and 12D patterns used within fuego/pachi themselves?
Not sure what a fast way to do this would be. In lieu of a solution, here are some random thoughts:

A simple implementation would be to store all 69000/32000 lookup values in a dict. Given a board, loop over (x,y) and lookup the dict value for each one. Pro: easy to implement. No transformations required. Con: wasted space, CPU-bound computation.
To overcome wasted space, as you suggested, decide on a 'representative' instance of each pattern. Do this by ordering all transforms of a position and taking the least (lexicographically or by hash, e.g.) of each as the representative. Pro: less wasteful of space. Con: extra lookup time for each pattern.
Note that CPU-boundness may not be too bad. Imagine that we put these lookup dicts onto the GPU.. you'd still have to compute liberties and send the tensor to the GPU in relatively small batches. I'd expect that the communication overhead is not worth the parallelism.
I see no real reason to expect a hash table/dict to be any slower than, say, a list indexed by the lexicographic rank order of patterns. In general that might make sense if patterns are accessed in a predictable order, since you could better use your CPU caches. But I wouldn't expect a pattern to the order in which 3x3-patterns are seen, e.g...
Having spelled this all out, I am now leaning towards the simple implementation, using a dict for each LUT...

fuego: only uses handcrafted 3x3 patterns and no 12D as far as i was able to determine, only stone positions are used no liberty counts.
pachi: i am still figuring out how they implemented it. seems by default they are not used but pachi is able to generate 3x3 patterns. http://pachi.or.cz/pat/ has some libraries people can download (~102000patterns) but i do not know if rotational & colour symmetries are included.. 12D patterns (and several others as well) are available but i have found no libraries for them
@oriskunk maybe you know more about pachi pattern implementation?
i am also in favor of the simple implementation, i think the additional space outweighs the computation time of any rotation before lookup and RL is CPU bound in AG as well.
Thanks! your summary helped with some other things as well. going to redesign the 12D to be able to use direct (x,y)liberty count lookup as wel.

is there a specific reason to use -1/0/1 for white/empty/black? i would like to change them to 0/1/2 ( 0 = empty ) it would be rather useful when creating pattern values.
i made an initial commit with model and my idea on (preprocessing structure)[https://github.com/MaMiFreak/RocAlphaGo/blob/debug-rollout/AlphaGo/preprocessing/preprocessing.py], my thought is to hold pattern dict in Gamestate and to use preprocessing as it is.

-1/0/1 is convenient for the value net among other things. I doubt +1 will have a big impact on lookup

the new value net is relative to active player so it does no longer use it but even so a single position evaluation with value net would encounter this -1 a maximum of 361x per position.
a single fast rollout would have to do this 5500 times on an empty board for the 3x3 & 12d patterns, nakade patterns not included and then for 300/400 moves.
right now the impact does not matter but at some point these small improvements will become important=)
i will use the +1 lookup

performed a successful first test with 3x3 patterns. managed to archief 10.5%acc but with a very limited dataset.

12d patterns are needed before i can test any game performance, right now it is unable to create any living group at all..
peformance is poor, have to check wheter it is my quick&dirty feature implementation or the 4500 one-hot tensors. (right now rollout move generation takes longer than SL move generation on cpu)
Nakade patterns are still a problem, i have not found a way to hash them in any way or to get 8000 of them...


New label: Patchomator

Created label for Patchomator (see https://github.com/Mac-Nerd/patchomator).

Thank you!


Collecting geometry from file instead of taking from STAC service metadata

Collecting geometry from file instead of taking from STAC service metadata

@vconrado Maybe, we can also remove the StacItemGeometry class in this new version. I can't see any other function using it.


Case sensitivity #include "Arduino.h"

I think the A in  #include "Arduino.h" should be an uppercase letter. In my case I had to change this, to make the code compile.

Ok, maybe it's a Arduino IDE behavior (I never use Arduino IDE). Anyway, I fixed. Thanks!

Thank you!


Brainhack Proceedings – an open & free journal for Brainhack project outcomes

Project info
Title:
Brainhack Proceedings – an open & free journal for Brainhack project outcomes
Project lead:
We are a team of people leading different parts of the process :)
Project collaborators:

Agah Karakuzu
Anibal Sólon
Cameron Craddock
Isil Bilgin
katja heuer

Matteo Mancini

 Pierre Bellec

Peer Herholz
Roberto Toro
Samuel Guay
Sofie Van Den Bossche

Registered Brainhack Global 2020 Event:
We are probably registered all over the place... including Ankara, Marseille, Ontario, NYC and many more 🙃
Project Description:
We are building the Brainhack Proceedings Journal – an open free publishing platform for the community to publish Brainhack project outcomes. The workflow from project to publication is entirely based on GitHub and completely open and transparent. We have implemented a first complete workflow \ö/ and are looking for 2 types of community interactions now:
(1) Some hacking projects around the platform.
For the Latex experts with love for design :) During Brainhack, would you like to join us for a little hacking session to improve the visual design of our latex template underlying the final paper pdf? While users only need to work with markdown language, the magic behind the pdf is currently latex. If you feel you could give us a hand, please come join our team.
For the Javascript lovers with interest in GitHub actions :) While we have a first workflow set up, it would be amazing to check that the web template is working, and Github actions compile the forked papers, that links are maintaining the correct targets throughout the process, and that the svg and banner scale across all pages.
(2) Some first submission tests
Brainhack proceedings is in its experimental phase. The first complete workflow is ready to be used and tested by all the amazing project outcomes from Brainhack Global: We invite everyone who would like to publish some original work coming out of the Brainhack, to come and test our workflow. Which will be the first publication featuring on the Journal website? 😃
Come and fork our template repo, fill your content, formulas and figures into the report.md file and see it magically transform into a paper preview. We have set up nice tutorials to take you by hand throughout the entire process.
Link to project repository/sources:
https://github.com/brainhack-proceedings/brainhack-proceedings.github.io
Goals for Brainhack Global 2020:

test workflow, all Github actions pass
test all links and check they are all linked to the submitted project
adapt the latex template to match the web-version style of the final paper
encourage some projects to actually submit their work :)

Good first issue
Please fix all links see this issue
Good second issue
Check that the svg banner scales correctly based on window size see this issue
Skills:

latex and love for design
some basics in Javascript and html
basic markdown knowledge to replace our template with your project content & submit to test the workflow and publish your work :)

Tools/Software/Methods to Use:

Nothing needs to be installed to use and test the workflow. It can be followed entirely online on GitHub.
simple local server to run the journal website locally and test changes for the svg banner

Communication channels:
Join us on Mattermost

Project labels

[x] I added all of the labels I want an associate to my project

Project Submission
Submission checklist
Once the issue is submitted, please check items in this list as you add under ‘Additional project info’

[x] Link to your project: could be a code repository, a shared document, etc.
[x] Goals for Brainhack Global 2020: describe what you want to achieve during this brainhack.
[x] Flesh out at least 2 “good first issues”: those are tasks that do not require any prior knowledge about your project, could be defined as issues in a GitHub repository, or in a shared document.
[x] Skills: list skills that would be particularly suitable for your project. We ask you to include at least one non-coding skill. Use the issue labels for this purpose.
[x] Chat channel: A link to a chat channel that will be used during the Brainhack Global 2020 event. This can be an existing channel or a new one. We recommend using the Brainhack space on Mattermost.

Optionally, you can also include information about:

[ ] Number of participants required.
[ ] Twitter-sized summary of your project pitch.
[ ] Provide an image of your project for the Brainhack Global 2020 website.

We would like to think about how you will credit and onboard new members to your project. If you’d like to share your thoughts with future project participants, you can include information about:

[ ] Specify how you will acknowledge contributions (e.g. listing members on a contributing page).
[ ] Provide links to onboarding documents if you have some:

Fantastic! Thank you so much for adding it in Katja! All looks good! going to publish now :hugs: Also added a screen shot to go with as image for now but apparently we are also missing a nice logo that might be part of the hackathon too indeed 🤗

Do you like our favicon ? I would like to use this as a logo.
And for the image, i was just about to add the link form the repo to the pattern

:) but happy if you want to go with something else :)

the screenshot you added is great! thanks so much @complexbrains :) <3

I can change, but I need png :( for that. Let me know if you want me to change.

no no, it looks great <3 Thanks a lot!!


Avoiding having both "start_instance" and "ip" set.

As the title.
Adding tests
Related to https://github.com/ansys/pymapdl/issues/2910

Codecov Report
Attention: Patch coverage is 92.85714% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 64.90%. Comparing base (ea42836) to head (e0f1305).
Report is 2 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff             @@
##             main    #2912       +/-   ##
===========================================
- Coverage   86.59%   64.90%   -21.69%
===========================================
  Files          48       48
  Lines        9449     9455        +6
===========================================
- Hits         8182     6137     -2045
- Misses       1267     3318     +2051

I'm not going to wait.
@pyansys-ci-bot LGTM


Request for longer response to make cache test more robust.

Why are these changes needed?

Avoid cache test failure like this one: https://github.com/microsoft/autogen/actions/runs/8312062884/job/22746520649?pr=2040
Related issue number

Checks

[ ] I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
[ ] I've added tests (if relevant) corresponding to the changes introduced in this PR.
[ ] I've made sure all auto checks have passed.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 34.87%. Comparing base (00972a6) to head (160faa3).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2043      +/-   ##
==========================================
- Coverage   36.90%   34.87%   -2.04%
==========================================
  Files          66       66
  Lines        7031     7031
  Branches     1538     1538
==========================================
- Hits         2595     2452     -143
- Misses       4210     4365     +155
+ Partials      226      214      -12

Flag
Coverage Δ

unittests
34.87% <ø> (-2.04%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


What Should Happen Instead?
addon installed
Reproduction Steps
inspection-report-20221011_192847.tar.gz
microk8s enable kube-ovn --force
Introspection Report
attached tar.gz
regards

With MicroK8s v1.25.4 revision 4214 on Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-52-generic x86_64) I get the kube-ovn installed and calico uninstalled. All fine so far - but kube-ovn crashes in the controller.
I1115 17:02:58.382743       7 controller.go:450] Starting OVN controller
I1115 17:02:58.382841       7 election.go:50] waiting for becoming a leader
I1115 17:02:58.382885       7 leaderelection.go:248] attempting to acquire leader lease kube-system/ovn-config...
I1115 17:02:58.466548       7 leaderelection.go:258] successfully acquired lease kube-system/ovn-config
I1115 17:02:58.466624       7 election.go:76] new leader elected: kube-ovn-controller-7f7b65ff49-pv9l8
I1115 17:02:58.466683       7 election.go:59] I am the new leader
I1115 17:03:03.379674       7 ovn-nbctl.go:1803] start ovn-nbctl daemon
E1115 17:03:03.382111       7 ovn-nbctl.go:1810] failed to kill old ovn-nbctl daemon: ""
E1115 17:03:03.382130       7 controller.go:58] failed to start ovn-nbctl daemon exit status 1
I1115 17:03:03.383372       7 controller.go:460] Waiting for informer caches to sync
E1115 17:03:03.391915       7 pod.go:1044] namespace kube-system network annotations is nil
E1115 17:03:03.391944       7 pod.go:97] failed to get pod nets namespace kube-system network annotations is nil
E1115 17:03:03.527206       7 ovn-nbctl.go:1854] failed to access ovn-nb from daemon, ""
W1115 17:03:03.527236       7 controller.go:65] ovn-nbctl daemon doesn't return, start a new daemon
I1115 17:03:03.527246       7 ovn-nbctl.go:1803] start ovn-nbctl daemon
W1115 17:03:03.668109       7 ovn-nbctl.go:48] ovn-nbctl command error: ovn-nbctl --timeout=60 --no-wait set NB_Global . options:use_ct_inv_match=false in 184ms
F1115 17:03:03.668418       7 controller.go:476] failed to set NB_Global option use_ct_inv_match to false: failed to set NB_Global option use_ct_inv_match to false: , "signal: illegal instruction (core dumped)"
may someone have an advice where to start digging ?

The "signal: illegal instruction (core dumped)" portion of the logs seem to indicate that kube-ovn is using some instructions that are not available for the underlying hardware and are causing a SIGILL exception? I am not sure how one could tackle this issue. Perhaps opening an issue on kube-ovn directly would get more traction?

The original issue was that KubeOVN had not been updated for 1.25, which was resolved with https://github.com/canonical/microk8s-core-addons/pull/115

The "signal: illegal instruction (core dumped)" portion of the logs seem to indicate that kube-ovn is using some instructions that are not available for the underlying hardware and are causing a SIGILL exception? I am not sure how one could tackle this issue. Perhaps opening an issue on kube-ovn directly would get more traction?

Thx, good hint. it looks like my CPU doesn't support avx512 - same issue as with "Kube OVN Charm" on some AWS systems. seems we need the same solution as in "https://bugs.launchpad.net/charm-kube-ovn/+bug/1989363/comments/5".

I was not aware of the -no-avx512 images. Would you mind testing them out to see if that is a solution?
Changing the failing deployments should be enough. If you find that this solves the issue, we can look into updating the addon to do such a check.

After changing the deployment and daemonset images to kubeovn/kube-ovn:v1.10.4-no-avx512 all is up and running. I didn't test the config now, but pods not crashLooping any longer.
Maybe when enabling kube-ovn we should ask if we just use the default or non-avx if the system check is too complex.
 Indeed, we'll work on improving this, if not with automated checks, at least a large note in the documentation and a flag when enabling the addon.
Thanks again!


Adds PipeBufWriter that buffers small writes

This PR is based on my master branch and adds PipeBufWriter that buffers small writes. It will try to send out data after every write in non-blocking mode to avoid adding latency and will block when buffer is full. To preserve performance with writes bigger than the buffer it will fall back to blocking behavior of PipeWriter.
This improves performance for small writes but may concatenate written data and requires call to .flush() when done writing.
I have added benchmark for this and also for case when PiperWriter is wrapped in BufWriter. Apparently Rust does a good job in optimizing this as it is as fast if not faster than PipeBufWriter but it may add unnecessary latency in comparison as it does not to try to opportunistically send.

I combed through the rebase and merged it as 96ea658d98fab66d471fc646405d4e12d15729d8, feel free to submit additional PRs if you want to augment (or take issue with) any of my changes.


10378/tf27

Proposed changes:

closes #10379
updated temp_keras_model + our sequence generator with changes from tf 2.7

replaced keras dependencies with tf.keras/ tf.python

removed seemingly unused dependencies (e.g. tf probability)

Progress

[x] need fix/workaround for this issue

passing a tensor with the attention mask instead of numpy array did the trick

[x] investigate differences in embeddings

[x] decrease tolerance successively to check how much differences there are (just as a test)
[x] investigate which configurations are affected and now much (compared to tf 2.6)
[x] check for api changes / issues that could explain differences
[x] run regression tests afterwards to check impact

[x] switch to 3.9 for tests (cf. PR by Alex) (see #8762)
[x] fix sentencepiece dependency

[x] figure out how to properly mark it as optional / transformer related (seems that must be added as main dependency)

[ ] inspect timings and memory consumption in comparison to tf 2.6

[x] created datadog notebook (10378/tf27)
[ ] investigate findings via profiler
[ ] check changelog for explanation

Summary
tf.python

Note: tf.python is to be used for private stuff and not public use. Ideally, we should remove those references if possible (e.g. those also appear in modules not touched in this PR such as layers).
for data generators, the keras export does not work well (the adapters ALL_ADAPTER_CLS in tf's data_adapter won't recognise it, leading to ValueError: Failed to find data adapter that can handle input: <class 'rasa.utils.tensorflow.data_generator.RasaBatchDataGenerator'>,)
the tf_util needed intemp_keras_model only exists in tf.python... (same function as in non-temp tf model :)

transformers <-> python 3.9
sounds like we should be able to use non-dev transformer version with 3.9 e.g. here
Overview: transformers versions

4.x versions tested caused trouble because of  'tuple' object doesn't have attribute as_list in 4.12.2 -> issue and potential fix already exists

issue description sounds like 4.6.1 isn't affected but it is :/
-> was resolved by the workaround with the tf.tensor
we end up with the same issue as with 3.5, i.e. differences in logits (more details below)

3.5: tests fail cause of slighty larger differences in tests now, e.g.          # Look at the value of first dimension for a few starting timesteps
          assert np.allclose(
                computed_sequence_vec[: len(expected_sequence_vec[index]), 0],
               expected_sequence_vec[index],
               atol=1e-4,
          )
          E           assert False
          E            +  where False = <function allclose at 0x7f347e38c320>(array([-0.0345806, -0.0533224]), [-0.03382749, -0.05373593], atol=0.0001)
          E            +    where <function allclose at 0x7f347e38c320> = np.allclose

( more details below )
>=2.4,<2.12:  is what we used before

depends on tokenizers 0.7.0 for which there is no python 3.9 build of

Differences in Embeddings

curent vs updated version

details: https://github.com/RasaHQ/rasa/tree/10378/tf/local/test-tf27
compared (locally):

transformers 4.13, tf 2.7.0 and python 3.9.9
current version tf 2.6.1, transformers 2.11, python 3.8.10

computed:
maximum over all element-wise absolute differences of the (first 5 dimensions of the) actual and expected embeddings (computed as in test_lm_featurizer_shapes_in_process_training_data)
computed the differences between these deviations using 1. and 2., respectively (per text)
result:

for all configurations != {gpt, xlnet} the results stay roughly the same, i.e. the maximum deviation between expected and actual outcome increases by only  < 1e-7
for gpt, the max. deviation increases by 1e-3
for xlnet, the max. deviation increases by 1e-2

Note: the differences between expected and actual embeddings are also > (precision of the hard coded expected embeddings) in the current version (1.)

same as 1. without attention mask

tested the same as above, with attention_mask commented out in the rasa versions of both environments to check whether it has something to do with that (cause there are some issues related to attention on hugginface repo) but ...
that yields larger differences for even more models :)

other

tokenizers (and our inputs) seem to be ok (i.e. verified that test case for gpt yield the same token ids in both versions), as to be expected 😅

/modeltest
include:
  - dataset: ["all"]
     config: ["all"]

More comparisons can be found in the corresponding datadog notebook (not dashboard) named "1037-tf27".  -- We're continuing to look into (fixing) the funny GPU mem/utilisation measurements, but from the time comparison (also in datadog, see notes above last two tables) and the scores (comment above), I expect no surprises (let me know if you do) and so I'll mark the PR ready for review to not delay this more later.

for data generators, the keras export does not work well (the adapters ALL_ADAPTER_CLS in tf's data_adapter won't recognise it, leading to ValueError: Failed to find data adapter that can handle input: <class 'rasa.utils.tensorflow.data_generator.RasaBatchDataGenerator'>,)

This caught my eye because it reminded me of a discussion from last upgrade (to 2.6), so noting here for mine and Daksh's benefit. We thought we had to make this change during upgrade to TF 2.6, but then we weren't able to reproduce the original error so reverted the change. This time I was able to reproduce:
ValueError: Failed to find data adapter that can handle input: <class 'rasa.utils.tensorflow.data_generator.RasaBatchDataGenerator'>, <class 'NoneType'>

We're continuing to look into (fixing) the funny GPU mem/utilisation measurements

So we were looking into if we could increase the sampling rate on the Datadog agent to try and capture more granular changes if they existed. From looking at the docs it is possible for us to control the sampling rate, but it's unclear what our limits are (most of the actual hard rate limits are defined in terms of events/hour/organisation - so not clear what's the maximum we could push any particular agent to). So we could just start guessing values but it might take an arbitrary amount of time to actually find anything, or we may find nothing at all - so we're not trying this approach any longer.

@koernerfelicia @dakshvar22 , as expected, I couldn't reproduce any unexpected differences between the two versions via GCP (some results are here). Only the inference times seemed a bit odd -- in both versions -- but that should get better when this is fixed. Let me know if something is still missing

some results are here). Only the inference times seemed a bit odd

@ka-bu Inference time results aren't in that folder or the notebook. Can I see what kind of odd behaviour you are seeing? I mean is it odd between TF 2.6 and TF 2.7? Asking because this issue will fix both TF 2.6 and TF 2.7 and not just one of them.

some results are here). Only the inference times seemed a bit odd

@ka-bu Inference time results aren't in that folder or the notebook. Can I see what kind of odd behaviour you are seeing? I mean is it odd between TF 2.6 and TF 2.7? Asking because this issue will fix both TF 2.6 and TF 2.7 and not just one of them.

Sorry for the late reply -- Yep the strange behaviour between tf 2.7 and tf 2.6 --- and also between different regression test runs for the same tf2.6 version -- can be seen in the notebook: very last table focusing on carbon-bot data only shows strange drops of maximum gpu usage
Regarding inference: For memory consumption, I can generate corresponding plots for inference (running with GPU) tomorrow. Didn't want to create spam but doesn't hurt to check that as well.  For the timings, there's the regression results above + measurements in datadog that are consistent across runs imho.

Two tests failed because of connection timeouts / installation problems. The rest looks fine. Any other comments/questions, or can we merge? :)

I hope to take a final look in an hour 😬

If this is a model breaking change (are models trained with a previous version of Rasa Open Source loadable with this new version, without re-training of course?) then that should be mentioned in the migration guide and changelog.

Thanks for the reminders and sorry for nearly skipping this 🤦♀
Loading and rasa shell-ing a model trained with the version from main works without problem (also with LanguageModelFeaturizer in the pipeline).
But rasa train-ing that model again (with the version from this branch) triggers retraining. @usc-m , can you confirm this? No idea why we would not load from the cache then...

No idea why we would not load from the cache then…

Because the features from the featurizers would change right?

On 28. Jan 2022, at 12:06, Kathrin Bujna @.***> wrote:
If this is a model breaking change (are models trained with a previous version of Rasa Open Source loadable with this new version, without re-training of course?) then that should be mentioned in the migration guide and changelog.
Thanks for the reminders and sorry for nearly skipping this 🤦♀
Loading and rasa shell-ing a model trained with the version from main works without problem (also with LanguageModelFeaturizer in the pipeline).
But rasa train-ing that model again (with the version from this branch) triggers retraining. @usc-m https://github.com/usc-m , can you confirm this? No idea why we would not load from the cache then...
—
Reply to this email directly, view it on GitHub https://github.com/RasaHQ/rasa/pull/10459#issuecomment-1024109598, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCOBGJBID42CYFGFCTPWDLUYJ2EXANCNFSM5JOBDFEA.
Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.

Because the features from the featurizers would change right?

Ohh but the data wouldn't have changed, so it should actually skip training. Iteresting edge case for the caching feature 😂

Because the features from the featurizers would change right?

Ohh but the data wouldn't have changed, so it should actually skip training. Iteresting edge case for the caching feature 😂

No you're right I think, because the input for e.g. DIET specifically changed and because we: https://github.com/RasaHQ/rasa/blob/413ac0551721709ca78dc1e890e075a75fa35533/rasa/shared/nlu/training_data/message.py#L140

Looks like Poetry bumped us to sanic 21.12.* when I was resolving the last lock file issues which appears to have a lot of breaking changes for us - investigating if we can pin this to the last working version to get this merged


Better memoization & minor fixes

Notable changes:
Added lodash as a dependency. I think we might as well have it. I tried importing the modules seperately, but ran into a lot of typing issues.
I improved the memoization and removed instrument, as it was not used except for the cache key.
Cleaned up a few things
Closes #440

Are we good if tests pass?

yes we are


Settings UI doesn't display content

The Problem
No content in Settings UI shown. Original message from https://github.com/microsoft/PowerToys/issues/4000#issuecomment-747294259, not understood why https://github.com/MicrosoftDocs/windows-uwp/issues/2869 was closed, so created this one.
ℹ Computer information

PowerToys version: 0.27.1
PowerToy Utility: ???
Running PowerToys as Admin: Yes
Windows build number:

📝 Provide detailed reproduction steps (if any)

Install https://github.com/microsoft/PowerToys/releases/tag/v0.27.1
Running PowerToys with admin rights
Right click on tray icon -> Settings
Nothing happens
Again, Right click on tray icon -> Settings
Settings UI loads, no content shown
7.Click on X (Close) -> Nothing happens
Again, click on X (Close) -> windows closes

✔ Expected result
I expect content in the settings UI, to adjust the several tools
❌ Actual result
Settings UI open unexpected (always on second try as described above) and doesn't show content.
📷 Screenshots

@Taneeda
did you set Run this program as an administrator yourself?

This is not how PowerToys is supposed to be executed with elevated privileges.

No, this was set, I think by the PowerToys Installer!?

@Taneeda

No, this was set, I think by the PowerToys Installer!?

The PowerToys installer doesn't set that flag and it should not be set, so please unckeck it.

@Taneeda what graphic card do you have?  an Nvidia based one?

@Taneeda did you have this issue prior?

my gut says this is the actual issue: https://github.com/microsoft/PowerToys/issues/3407.  If this is, there could be an nVidia setting we can toggle off.

I see you're still using Win version 18-03. Is it possible that has something to do with it?

@Taneeda

No, this was set, I think by the PowerToys Installer!?

The PowerToys installer doesn't set that flag and it should not be set, so please unckeck it.

I tried, unfortunately no change, still the same empty settings UI

@Taneeda what graphic card do you have? an Nvidia based one?

@Taneeda did you have this issue prior?

Yes, since 0.17 I think. Meanwhile I tried several versions, but always the same behavior

I see you're still using Win version 18-03. Is it possible that has something to do with it?

Hopefully not :) windows update are managed by my company from the IS department and major windows updates are not performed very much :D

Well - regardless if this is the cause of not - you could ask your IS department if you could have an update for Christmas? 😄

Ooo, I missed the 1803. Good catch!  @enricogior that would make this 243 then, webview not rendering.

@crutkas @Taneeda
if turning off Run this program as an administrator in the PowerToys.exe properties doesn't make any difference, this is https://github.com/microsoft/PowerToys/issues/243


Async serial uart read

serial uart read for embassy async
Tested on ESP32C3 only.
NOTE: embassy_hal_async does not have a Read trait for serial, only Write traits, so I could not follow the same implementation pattern as in the write case. The new function is called read, not sure if it is Ok, there was not a read function for nb implementation. If you think is to generic, I can change it to read_async
Also fix an issue with async write on large buffers, was panicking with Err Would Block. Discovered when testing reading and echoing back a large reader buffer.
NOTE: this is a reopen of the #615 PR.
Must

[x] The code compiles without errors or warnings.
[ ] All examples work.
[x] cargo fmt was run.
[x] Your changes were added to the CHANGELOG.md in the proper section.
[x] You updated existing examples or added examples (if applicable).
[x] Added examples are checked in CI

Nice to have

[x] You add a description of your work to this PR.
[x] You added proper docs for your newly added features and code.

Could you address the remaining review comments from https://github.com/esp-rs/esp-hal/pull/615 here please.

I did address them in commit https://github.com/esp-rs/esp-hal/pull/620/commits/6fd1f346bb20b9bc818aa52bb3f5a82cf8944abc
Also update the PR number in latest commit https://github.com/esp-rs/esp-hal/pull/620/commits/e38de414e949256120c4e5cea294918ba46bab53

Sorry, you are right I lost some code on commit. I will put it back


[Bug]: Size Frequency section in data_boot_001.ss and data_expval.ss identical to input file

Describe the bug
While testing PR #540 to fix the size frequency method in CompReport.sso, @iantaylor-NOAA saw that the size frequency section in data_boot_001.ss and data_expval.ss are identical to the input file (but were different in 3.30.19).
It looks like this commit https://github.com/nmfs-ost/ss3-source-code/commit/61d2cba7ea9ce1d640920ec242c54098f039443e, which included a variety of other changes, may have inadvertently led to the observed output getting written to all three data file types.
In particular this change looks incorrect: https://github.com/nmfs-ost/ss3-source-code/commit/61d2cba7ea9ce1d640920ec242c54098f039443e#diff-17a83800a032b2061d3ddd5c6da9d766d9796ecb81f2ebf1599b9808c206247eL775-R782
but the change includes moving from looping over size frequency observations to size frequency methods, so I'm assuming the fix is not as simple as swapping SzFreq_obs and SzFreq_exp.
Lesson from all this: more tests related to generalized size frequency and/or bootstrap data would be helpful.
The one test model with generalized size frequency data has the same scale and units for both size frequency methods: https://github.com/nmfs-ost/ss3-test-models/blob/main/models/two_morph_seas_areas/data.ss#L455. And I don't think we have any tests for expected values or bootstrap data.
To Reproduce
See PR #540.
Expected behavior
Size frequency section in data_boot_001.ss and data_expval.ss should be different from the input file.
Screenshots
No response
Which OS are you seeing the problem on?
No response
Which version of SS3 are you seeing the problem on?
Versions after v.3.30.19
Additional Context
No response

@Rick-Methot-NOAA, should we consider a bug fix release once this issue is resolved?
Yohei Tsukahara is interested in bootstrapping for Pacific bluefin tuna.
I don't know if there are other issues that could be rolled in as well.

Just created a <IP_ADDRESS> milestone

I looked at the code and see how an errant cut 'n paste involving:
https://vscode.dev/github/nmfs-ost/ss3-source-code/blob/main/SS_write_ssnew.tpl#L398-L401
created this problem.  Should be straightforward to restore the expval and boot code.


[ts-styled-plugin] colon expected

I noticed a bug recently. I got an error like this:

The code is available here: https://gist.github.com/ApacheEx/e1242d762a3fc98926d1605f1e8c6e77
My Environment:

Editor: VSCode 1.23.1
Extension: vscode-styled-components 0.0.18
OS: macOS Hight Sierra 10.13.4

I have a similar problem maybe is related, it happens in styled-components as well

Another example when using styled-components:

[ts-styled-plugin] colon expected

Here's the code that controls this: https://github.com/Microsoft/typescript-styled-plugin/blob/f82699d1a0027cb850118adfcdd8cf88203573dc/src/index.ts#L20
And steps to contribute: https://github.com/Microsoft/typescript-styled-plugin#contributing


Sparse KV cache design

Design doc - https://bytedance.larkoffice.com/wiki/CiW4wA3etijfWHkN8rWctNOQnxg

Design doc - https://bytedance.larkoffice.com/wiki/CiW4wA3etijfWHkN8rWctNOQnxg
https://bytedance.larkoffice.com/wiki/ShXJwPtyEiy9pKkVr0EcurS0nSg

could you please open this design doc, which we don't have access, thanks

Design doc - https://bytedance.larkoffice.com/wiki/CiW4wA3etijfWHkN8rWctNOQnxg
https://bytedance.larkoffice.com/wiki/ShXJwPtyEiy9pKkVr0EcurS0nSg

could you please open this design doc, which we don't have access, thanks

https://docs.google.com/document/d/13_cpb31P9VOmPGa_tZ70s7z1vXGP_UenXf1WVuIppCk/edit#heading=h.evhjclx158o3 Thanks for your interest. The updated doc is pasted in the google doc.


refactor code to unify the format of ajax request

Current Behaviour: Currently, in the project, there are multiple ways being followed to perform ajax request to the server. For eg., Dashboard.jsx uses fetch, ViewData.jsx uses vanilla JS' XMLHttpRequest().
Expected: One format should be followed throughout the project.
Remarks: I am working on it.

@orionpax00 @sumedhe need your suggestions regarding this. I would say, axios or fetch would be better alternatives.

I have changed the format to fetch in ViewData.jsx you can do the rest:)


No How To Article

Hi, Can you please write a how to article for the Nearby Sharing protocol too?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 871cb1e7-6c1b-af8d-f376-e25cd243200e
Version Independent ID: 28310262-583a-a333-b936-a4675e089ca4
Content: Nearby sharing - Universal Windows Platform
Content Source: project-rome-docs/nearby-sharing/index.md
Product: windows
GitHub Login: @PatrickFarley
Microsoft Alias: pafarley

Currently there is a sample in the SDK for Android for sending a file or link, that seems to work.
I'm wondering whether the opposite is also possible?

Thanks for the feedback and thanks jkommeren for pointing that out! I've added a link to the SDK sample from this page. We don't have a how-to to publish at this time, but we will take the feedback into account!


⚠ Bofrost.lu has degraded performance

In 4e24b02, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 1405 ms

Resolved: Bofrost.lu performance has improved in b833f65.


[POC] Embedding MakeCode for programming output micro:bit

Preview url: https://review-ml.microbit.org/embed-makecode/

Closing PR as changes are merged into poc branch (https://github.com/microbit-foundation/ml-trainer/pull/264), will keep branch for now


Incorrectly generated latex file

Why can't I generate a paper after running the code, there are only some pictures and titles, like the abstract, introduction, etc. There is no content in the paper.

Please see the FAQ:
Why has a PDF or a review not been generated?
The AI Scientist finishes an idea with a success rate that depends on both the template, the base foundation model, and the complexity of the idea. We advise referring to our main paper. The highest success rates are observed with Claude Sonnet 3.5. Reviews are best done with GPT-4o, all other models have issues with positivity bias or failure to conform to required outputs.

Please see the FAQ:
Why has a PDF or a review not been generated? The AI Scientist finishes an idea with a success rate that depends on both the template, the base foundation model, and the complexity of the idea. We advise referring to our main paper. The highest success rates are observed with Claude Sonnet 3.5. Reviews are best done with GPT-4o, all other models have issues with positivity bias or failure to conform to required outputs.

My model, deepseek-coder-v2-0724, displays the following error. Is It related to this error?
Response Content: {"totalHits":0,"limit":10,"offset":0,"results":[]}
Error: string indices must be integers, not 'str'
Response Status Code: 200

Haven't seen that, can you uncomment the try catch and see where the error comes from?


Repository includes non-free fonts

See https://github.com/typst/packages/issues/499

Thanks again for your feedback, I have removed the direct link to the fonts.zip in README.md. I will close this issue later.


add days it takes to close a pr to repos.pulls

Signed-off-by: Mike Ng<EMAIL_ADDRESS>@chenz4027 @harryli256
an example of how to update the db entries with new field

which allows you to do queries like
db.repos.pulls.aggregate([{$match: {"repo_name": "_put_repo_name_here_"}}, { $group: { _id:null, averageDaysToClose: { $avg: "$days_to_close" } } } ])

which can be market it as "prediction" of how long a new PR will be close.

or even better just do this which gives you a repo per repo_name
db.repos.pulls.aggregate([{"$group":{"_id":"$repo_name","averageDaysToClose":{"$avg":"$days_to_close"}}}])