Fine-tuning on customized dataset

Hi, thanks for the nice work! I'm trying to fine-tune PSALM on customized dataset, I wonder what specific modifications are needed (e.g., how should I organize data format, data loader etc.)? For example, if I want to further fine-tune on gRefcoco, can current gRefcoco evaluation dataset directly be used for fine-tuning? Many thanks!

Hi@ys-zong
If you want to fine-tune on gRefcoco, you need to rebuild the data pipline:

You can follow datasets/build_RefCOCO.py to prepare gRefCOCO train data. Each sample in this should contain a sentence and several mask annotations
Following RefCOCO_dataset at here, build the gRefCOCO_dataset, it seems that you just need to change the IO of this dataset class
train model following refcoco pipline, provide data_dict['dataset_type'] = 'referring_coco' can make model train in refcoco pipline

Thanks for the instructions. I tried it but I guess the main problem is how to deal with the "no-target" samples in gRefcoco - anns field will be an empty list. If directly use RefCOCO_dataset, there will be error here during preprocessing. If I use gRefcoco_Dataset (main difference seems to be in this line), it gives error during calculating loss:
[rank0]:   File "/mypath/PSALM/psalm/model/mask_decoder/Mask2Former_Simplify/utils/matcher.py", line 208, in forward
[rank0]:     return self.memory_efficient_forward(outputs, targets)
[rank0]:   File "/opt/conda/envs/psalm/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/mypath/PSALM/psalm/model/mask_decoder/mask_criterion/pretrain_criterion.py", line 417, in memory_efficient_forward
[rank0]:     tgt_mask = targets[b]["masks"].to(out_mask)
[rank0]: TypeError: 'NoneType' object is not subscriptable
``

I wonder if the current model supports this? Similar to this, how did you handle/evaluate the no-target predictions during evaluation of gRefcoco? Did you evaluate the N-Acc defined in the the original paper? Thanks!

@ys-zong
Yes, you are right. We do not deal with no-target situation (annotation is empty) when training. May be you need to rewrite the dataset mapper
When we ft on gRefcoco, we add a all-zero mask in annotation. Here are my gRefCOCO dataset for training
class gRefCOCO_dataset(RefCOCO_dataset_pooling):
    def __getitem__(self, idx):
        mask_type = 'polygon'
        data = self.data[idx]
        image_file = data['image_info']['file_name']
        image_folder = self.data_args.refcoco_image_folder

        data_dict = {}
        data_dict['file_name'] = os.path.join(image_folder, image_file)
        data_dict['height'] = data['image_info']['height']
        data_dict['width'] = data['image_info']['width']
        data_dict['image_id'] = data['new_img_id']
        data_dict['annotations'] = data['anns']
        for annotation in data_dict['annotations']:
            annotation['bbox_mode'] = BoxMode.XYXY_ABS
            # annotation['category_id'] = self.coco_id_to_cont_id[annotation['category_id']]
            if annotation['category_id'] in self.coco_id_to_cont_id:
                annotation['category_id'] = self.coco_id_to_cont_id[annotation['category_id']]
            elif annotation['category_id'] in self.coco_id_to_cont_id.values():
                annotation['category_id'] = annotation['category_id']
            else:
                raise ValueError
            annotation['image_id'] = data['new_img_id']

        if len(data_dict['annotations']) == 0:
            # build negative mask
            annotation = {}
            annotation['bbox_mode'] = BoxMode.XYXY_ABS
            annotation['category_id'] = 1
            annotation['area'] = 0.0
            annotation['bbox'] = [0,0,0,0]

            binary_mask = np.zeros((data_dict['height'],data_dict['width'])).astype(np.uint8)
            mask_type = 'bitmask'

            segmentation = encode(np.asfortranarray(binary_mask))
            segmentation = {
                'counts': segmentation['counts'].decode('ascii'),
                'size': segmentation['size'],
            }
            annotation['segmentation'] = segmentation

            data_dict['annotations'] = [annotation]

        if isinstance(self.data_args.image_processor, dict):
            processor = self.data_args.image_processor['instance']
        else:
            processor = self.data_args.image_processor
        data_dict = processor.preprocess(data_dict, mask_format=mask_type)
        # instruction = data['instruction']
        sentences = data['instruction']
        # prefix_inst = 'Referring Segmentation according to the following instruction:'
        prefix_inst = 'This is an image <image>, Please doing Referring Segmentation according to the following instruction:'
        instruction = ''
        for sent in sentences:
            instruction += ' {}.'.format(sent['sent'])
        # instruction = 'Please segment all the items in this image'

        # num_class = len(self.coco_class_name)
        # category = '<cls>, ' * (num_class-1) + '<cls>.'

        if self.data_args.seg_last:
            sources = [[{'from': 'human', 'value': prefix_inst + '\n<refer>'},
                        {'from': 'gpt', 'value': '\nSure, the segmentation result is <seg>'}]]
        else:
            sources = [[{'from': 'human', 'value': prefix_inst + '\n<refer>'},
                        {'from': 'gpt', 'value': '\nSure, the segmentation result is'}]]
        # sources = self.preprocess_multimodal(copy.deepcopy(sources))

        text_dict = self.preprocess_llama2(sources, self.tokenizer)
        input_ids = text_dict['input_ids'][0]
        labels = text_dict['labels'][0]

        token_refer_id = self.preprocess_referring_instruction(instruction)
        refer_embedding_indices = torch.zeros_like(input_ids)
        refer_embedding_indices[input_ids == REFER_TOKEN_INDEX] = 1

        data_dict['input_ids'] = text_dict['input_ids'][0]
        data_dict['labels'] = text_dict['labels'][0]
        data_dict['dataset_type'] = 'referring_coco'

        data_dict['token_refer_id'] = token_refer_id
        data_dict['refer_embedding_indices'] = refer_embedding_indices
        return data_dict

You can try this

Thanks! I used this dataset class and it works well. I wonder what are the hyper-parameters you used for training (batch size, learning rate, epochs, etc.)? I did get improvement after fine-tuning but it's around 60 cIoU and 65 N-acc, which is much lower than yours.

Hi @ys-zong
Sorry for replying late
The training log and settings for ft gRefCOCO have been lost...., and I give you some points in my mind

ft epoch > 10
score threshold may be 0.7
lr is 4e-5

Hoping these are helpful to you and if you can share your training log, may be I can help you to reproduce my result

Thanks for your help! I can roughly reproduce the performance with some HP tuning.


Feature Request: Checking if current route has a specific middleware

Feature Request
Checking if current route has a specific middleware
I'm using the latter, and thought it would be nice to see in the package.
if (route().current().middleware('auth')) {
    location.href = route('home');
}

Instead of:
if (
    route().current('auth.login') ||
    route().current('auth.register') ||
    route().current('auth.forgot-password')
) {
    location.href = route('home');
}

Sounds like a great PR :)
I would merge this


Websocket over devtunnel is not working

I am trying to connect a websocket over the devtunnel but the connection is not fully established.
I am trying using the wss:// protocol over the https:// endpoint.
Are websockets currently supported?

Yes websockets are supported.
Can you share more information to help us resolve your issue?

I am running a simple java websocket server locally on port 8080.
My websocket client is postman for testing purposes.
I can connect to ws://localhost:8080 with no issues.
Also I am able to use ngrok for this port but I want to switch to using devtunnels.
For devtunnels I use: wss://TUNNELID-8080.use.devtunnels.ms.
This tunnel has port 8080 set to auto.
Should I explicitly add the wss protocol for port 8080?
When trying to connect over devtunnels, I get the callback for connectionEstablished in my ws server but nothing after that.
The connection never moves past Connecting in the postman client.
What am I missing?
What other details should I provide to help debug this issue?
Here is my verbose output:
HostSSH: Receiving #23 ChannelOpenMessage(ChannelType: client-ssh-session-stream, SenderChannel: 1) ClientSSH: SshServerSession:34817906 ConnectAsync HostSSH: Sending #23 ChannelOpenConfirmationMessage(RecipientChannel: 1)(SenderChannel: 1) HostSSH: SshClientSession:33592719 OnChannelOpenCompleted(SshChannel(Type: client-ssh-session-stream, Id: 1, RemoteId: 1)) ClientSSH: Sending #0 KeyExchangeInitMessage HostSSH: Receiving #26 ChannelWindowAdjustMessage(RecipientChannel: 1)(BytesToAdd: 4194343) HostSSH: Sending #26 ChannelWindowAdjustMessage(RecipientChannel: 1)(BytesToAdd: 4194343) ClientSSH: Local version: SSH-2.0-Microsoft.DevTunnels.Ssh_3.11, remote version: SSH-2.0-Microsoft.DevTunnels.Ssh_3.11 ClientSSH: Receiving #0 KeyExchangeInitMessage ClientSSH: KeyExchange negotiation: Server (diffie-hellman-group16-sha512, diffie-hellman-group14-sha256) Client (ecdh-sha2-nistp384, ecdh-sha2-nistp256, diffie-hellman-group16-sha512, diffie-hellman-group14-sha256, ext-info-c) => diffie-hellman-group16-sha512 ClientSSH: PublicKey negotiation: Server (ecdsa-sha2-nistp384) Client (rsa-sha2-512, rsa-sha2-256, ecdsa-sha2-nistp384, ecdsa-sha2-nistp256) => ecdsa-sha2-nistp384 ClientSSH: ClientEncryption negotiation: Server (aes256-cbc, aes256-ctr) Client<EMAIL_ADDRESS>aes256-cbc, aes256-ctr) => aes256-cbc ClientSSH: ServerEncryption negotiation: Server (aes256-cbc, aes256-ctr) Client<EMAIL_ADDRESS>aes256-cbc, aes256-ctr) => aes256-cbc ClientSSH: ClientHmac negotiation: Server<EMAIL_ADDRESS><EMAIL_ADDRESS>hmac-sha2-512, hmac-sha2-256) Client<EMAIL_ADDRESS><EMAIL_ADDRESS>hmac-sha2-512, hmac-sha2-256) =><EMAIL_ADDRESS>ClientSSH: ServerHmac negotiation: Server<EMAIL_ADDRESS><EMAIL_ADDRESS>hmac-sha2-512, hmac-sha2-256) Client<EMAIL_ADDRESS><EMAIL_ADDRESS>hmac-sha2-512, hmac-sha2-256) =><EMAIL_ADDRESS>ClientSSH: ClientCompression negotiation: Server (none) Client (none) => none ClientSSH: ServerCompression negotiation: Server (none) Client (none) => none ClientSSH: Receiving #1 KeyExchangeDhInitMessage ClientSSH: Sending #1 KeyExchangeDhReplyMessage ClientSSH: Sending #2 NewKeysMessage ClientSSH: Receiving #2 NewKeysMessage ClientSSH: Sending #3<EMAIL_ADDRESS>server-sig-algs=rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256<EMAIL_ADDRESS><EMAIL_ADDRESS>ClientSSH: SshServerSession:34817906 encrypted ClientSSH: Receiving #3<EMAIL_ADDRESS>server-sig-algs=rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256) ClientSSH: Receiving #4 ServiceRequestMessage(Service: ssh-userauth) ClientSSH: Sending #4 ServiceAcceptMessage(Service: ssh-userauth) ClientSSH: Receiving #5 AuthenticationRequestMessage(Method: none, Username: tunnel) ClientSSH: SshServerSession:34817906 Authenticating(Username: tunnel, Key: password) ClientSSH: ClientNone authentication succeeded. ClientSSH: Sending #5 AuthenticationSuccessMessage ClientSSH: Sending #6 PortForwardRequestMessage(RequestType: tcpip-forward)(AddressToBind: localhost, Port: 3000) ClientSSH: Sending #7 PortForwardRequestMessage(RequestType: tcpip-forward)(AddressToBind: localhost, Port: 5500) ClientSSH: Sending #8 PortForwardRequestMessage(RequestType: tcpip-forward)(AddressToBind: localhost, Port: 8080) ClientSSH: Receiving #6 SessionRequestSuccessMessage ClientSSH: Receiving #7 SessionRequestSuccessMessage ClientSSH: Receiving #8 SessionRequestSuccessMessage ClientSSH: Receiving #9 ChannelOpenMessage(ChannelType: forwarded-tcpip, SenderChannel: 0) ClientSSH: PortForwardingService forwarded channel #0 connection to <IP_ADDRESS>:8080. ClientSSH: Sending #9 ChannelOpenConfirmationMessage(RecipientChannel: 0)(SenderChannel: 0) ClientSSH: SshServerSession:34817906 OnChannelOpenCompleted(SshChannel(Type: forwarded-tcpip, Id: 0, RemoteId: 0))

Is the tunnel (or the specific port 8080) set to public?

Yes. Here are my details
% devtunnel show Tunnel ID             : {TUNNELID}.use Name                  :  Description           :  Tags                  :  Access control        : {+Anonymous [connect]} Host connections      : 1 Client connections    : 0 Current upload rate   : 2639 bytes/s (limit: 5 MB/s) Current download rate : 2415 bytes/s (limit: 5 MB/s) Upload total          : 88 KB Download total        : 66 KB Ports                 : 3   3000  auto  https://{TUNNELID}-3000.use.devtunnels.ms/     5500  auto  https://{TUNNELID}-5500.use.devtunnels.ms/     8080  auto  https://{TUNNELID}-8080.use.devtunnels.ms/   Tunnel Expiration     : 30 days

Issue got solved without any changes on my end. Also no devtunnels update.
Maybe a network issue solved a reboot.

I had the exact same issue but my server was running ASP.NET Core.
This is how I resolved the issue 👇
Faulty setup:

my websocket server application was running locally as https://localhost:7000
I created a devtunnel with the following port mapping  devtunnel port create -p 7000 --protocol https
Actual behaviour: the connection was never fully established and I've never received any data

Working setup:

I changed my local running websocket server application to http://localhost:6000
I created a devtunnel with the following port mapping  devtunnel port create -p 6000 --protocol http
Now everything was working

The switch locally from https to http solved the issue (apparently 😉).
Kudos to dev-tunnels 🤩💜


FT specific work dir settings

Hi,
I am using iron.nvim mainly for julia code. My problem is, that my neovim is in the repo root (ensured by nvim-rooter) and the source file is located for example in src/main.jl
If the Julia code includes relative imports (e.g. include('otherfile.jl')), then this file cannot be found because neovim is still in the repo root.
Is it possible to add a filetype specific option in iron.nvim to set the workdir before opening the REPL?

This does not work. Iron starts and exits immediately, I cannot even read the error message.

It works for me:

Do you miss the -i option?

No, I probably missed the -e
It works for now. I will close this after some testing. Thanks


Added getter for Http Kernel middleware

Hi.
We have run into a barrier, where we have a desire to get the global middlewares of the Http Kernel, yet it seems it is protected, and there is no method to extract it. This PR adds a simple getter.
I did notice there are already getters for the middlewareGroups property, so I figured it is safe to add the same API for the middleware property?
Since this adds a simple getter method, it is unlikely to have any breaking changes. The only breaking change I can imagine is if a project has extended the Http Kernel, and have a getMiddleware method with another signature.
I am however worried if the naming getMiddleware can result in confusion, as it only returns the middleware registered on the Http Kernel, and not middleware groups or route middlewares.

Try https://github.com/spatie/invade for now. I've added a getter for this in Laravel 11.


#81 - Configured and enabled rootless builder and runner

Introduction
From a security perspective, it is generally not a great idea to mount the docker.sock into containers, since that essentially allows the user's inside the containers (which either need to be root in order to interact with the socket, or a member of the same GID as the docker group on the host) to run workloads as root on the host. As such, it may make sense to instead use rootless Podman within the containers, such that Functionary can run entirely rootless and not rely on exposing the host's docker.sock to the containers. This way, since user's do not need to be root, the unprivileged user's in the containers are mapped to some higher non-privileged UID on the host, which reduces the possibility of a user from breaking out as root onto the host.
Introduced Changes

Separated the Functionary image into two different images: one with Podman installed, and another without Podman installed

Both images run as a non-privileged user functionary with UID 1000

The two containers that use the Podman-Functionary image are the build-worker and the runner-worker

Both containers will create a podman.sock that will be used in place of the docker.sock
These two containers are granted additional capabilities, which are required for rootless Podman to build and run containers.
They DO NOT use nor require privileged mode in order to function.

Added entrypoint.sh scripts to handle the launching of the different services. This allows us to perform additional configurations before launching a service if needed
Runner launched images now use v1.0.0 tag instead of latest, to prevent unnecessary pulling of the image

Set Up Instructions
On host: sudo apt install fuse-overlayfs
Push the necessary images to the local registry for the deployment:

Spin up Functionary
exec into the build-worker container
cd templates/
Run ./build.sh -p
Select whichever repository you want to pull the image down

Testing

Verify user in the container is not root

exec into one of the containers
Run whoami
Verify result is functionary

Verify user in container is not running as root on the host

exec into one of the containers, and run sleep 100
On the host, run ps -af | grep sleep
Verify that the user running that process is not root

Verify podman is installed in the build-worker and runner-worker containers

exec into both containers
Run podman info
Verify podman spits out a bunch of info about it's installation

Verify podman.sock exists in the build-worker and runner-worker containers

exec into both containers
ls -al ~
Verify socket exists

Verify podman can run containers in the build-worker and runner-worker containers

exec into both of the containers
Run podman run alpine echo hello world
Podman should pull the latest alpine image, spin up the container, and print out "hello world"

Verify build-worker can build and push images to the registry

Use functionary CLI to publish a package to functionary
View the logs of the build-worker, verify the build starts and succeeds
You can exec into the build-worker container and run podman image ls to verify image exists

Verify runner-worker can execute tasks

From the Functionary UI, launch a task
View logs of the runner-worker, verify the runner pulls the necessary image, creates the container, and runs the container
From the Functionary UI, verify the task status updates to Complete, and the result is expected

General notes:

Switch back to the slim image rather than using alpine
Let's focus on getting the builder to do the rootless podman thing and allow the runner to continue using the docker sock connection of the host.
The containers that don't interact with docker in any way can be updated to not run as root.  Let's make the UID a build arg though rather than hardcoding it to 1000.
The v1.0.0 tag change shouldn't be needed.  Let's just stick with latest.
The build worker shouldn't need the cap-add stuff or the fuse mount.
Let's verify the security_opts as well.
Some of what's happening in the podman.Dockerfile may be specifically in service of spinning up containers, which the build worker doesn't need to do.  We should try to streamline what we can for that context.
Let's do this for the Dockerfiles:

The runner uses the existing debug.Dockerfile (pick a new name for it)
The builder uses the new podman.Dockerfile
Everything else uses what is currently (in your branch) named functionary.Dockerfile

You don't need to mount in the template stuff into the containers

I'm going to close out this PR.  Some of what this PR was aiming for (non-root users in the containers where possible, a proper entrypoint script) were used as inspiration for changes in #104.  Having the builder and runner operate completely rootless is something that will be revisited later when we look to create cloud native iterations of those components.


Preprocessing video dataset. Using Mediapipe Pose solution.

Hello,
I don't know if we can ask general questions here but I'm giving it a go. Please close this if its not relevant.
I'm using Windows, Jupyter notebook, Mediapipe to collect keypoints, tensorflow, keras.
I am doing an action recognition task and I have a dataset with videos. I used Mediapipe to collect the keypoints through webcam. And now stuck at preprocessing the video dataset. So my question is:
--There are so many methods we use for video classification. One being Pose detection(using the keypoints of our body). Where can we learn about this specific method? Any websites, books, blogs, please let me know.
--Learning the theory is one and learning the code is another. Where can I find the basic syntax code for this particular method?
Any answer would be really appreciated.
Thank you.

Hi @Dilipraj6 ,
Please MediaPipe github code for your code reference.

Thanks for replying. I have checked the link which you've provided and I didn't find any code. I don't know if I checked at the right place in that link. Anyways, the thing is I have successfully got to a point where I can see the Mediapipe's landmarks and the drawings through my webcam. The actual help I need is for preprocessing my video dataset. I don't know how to do it. If you can help me in any way, that would be really good. It will help me a lot.

Hi @Dilipraj6 ,
Could you please refer this for code and theoretical understanding on pose estimation.


Some suggestions for improvement

First thanks for sharing this code! Can't wait to get it working on my system.
Some suggestions:  regarding the main.c
#define CONFIG_CALLSIGN "R2BDY"
#define CONFIG_LOCATOR4 "KO85"

Is it an idea to use a seperate config file for those?
if a gps module is used, is the locator variabele still needed? Lets asume i take the beacon with me to another location and the gps is working, as a clock but also as a gps, the location is availible as coordinates. On the another hand, i could imagine you don't want to have all code in the module to calculate the gps coordinates to the grid.

Regarding the issue https://github.com/RPiks/pico-WSPR-tx/issues/6
Suggestion to add the variabeles
gpio_set_function(uart_id ? 8 : 12, GPIO_FUNC_UART);
gpio_set_function(uart_id ? 9 : 13, GPIO_FUNC_UART);
Also in the main config/init file ?
Regards
Eelco

Hi Eelco,
Sorry for the belated answer, but I really had the hands full.
Thank you for the suggestions: they help to focus on necessary improvements.
A few weeks ago I added the console function to the core of this project (pico-hf-oscillator). The user can set what they need by enter simple commands such as SETFREQ 24915000. Other commands are available too.
This holidays I'm planning to implement similar function in this WSPR project.
So, I would like to address a question to you: what do you think about such console? What commands would you add to it? I possess my own view on this, but such alternative mention would be useful.
Please look at an example video in pico-hf-oscillator. Thank you in advance.
Cheers, Roman
R2BDY

Hey Roman, a console as alternative to do quick changes is an practical approach for some areas i mentioned. If we take a closer look to the needs, and time investments. Just being blund my backlog:

As a user i want to be able to have a wspr beacon that is flexible and can be used in a permanent location ans during holidays in other locations.
As a user i want to have the main settings permanent and be able to make changes in the configuration. Roel Kroes, has made this in his https://github.com/RoelKroes/TBTracker-RX project, main settings compiled, and changes by web interface, it is a bit like your console idea.
As a HAM i want to have a permanent stable signal (no drifting)  on the main bands with almost none harmonics.  This one demands possible a bit more details, and maybe external filters or modules. But lets give this one focus and make the demission what band would be the best candidate  (as an rookie i think 20 m band)
As a technition i would suggest always to set some variables permanent, in such a way the beacon will after startup always follow regulations as the default frequency used for WSPR
As a user i want to have some schema's and documentation and a logical approach to build the project (and this is something that asks time and energy and in most cases you do this after the POC is done:-)
For me as non electronics guy i need to understand the main approaches and get it in a working model. Like ok if i add an antenna, do i flollow the guideline for 20 half band diplodipole and connection to pens, and some examples,

I am waiting for my pico, i expect it today, i have a GPS module for my Roel Kroes projects here in the stash, i have a pi5 with the SDK on it and it compiled your code. Next steps is to build the device and see if i can receive the beacon in the Shack.
I have looked as non programmer in your pico hf project i like the idea  behind it, I would suggest to share the amplifier model, but because i am not that experienced. I was wondering will the pico be able to give a true stable basic frequency?  In most projects i have seen (arduino) they use external clocks or crystals.
I also used Meshtastic, they go even further on, and have a Bluetooth interface and web interface to make changes in the config, or have true interaction with the device with an app over Bluetooth.  But, it all depends time, and effort.
My goal would be, have the device stable in a watertight box and have a stable beacon others can use to check propagation, and or use the beacon, to report it as start for a conversation:-)
I hope my input will give you some ideas..
Keep you posted over my progress.


[5.4] Correct variable name

'Amount' is for continuous quantities, not discrete quantities. 'Number' is the correct variable name in the cases I've changed.
E.g. "I had a small amount of ketchup" - correct
"I adopted a large amount of dogs" - wrong, should be 'number' of dogs.

Agree, let's not propagate English mistakes.
By the way, would conflict with #20439.

Has conflicts.

Conflicts resolved.

You should also correct Arr::random().


Could you publish a version that will work for DOCTYPE html?

I'll look into it

Thanks
i try to make html without erroru: D (https://validator.w3.org/nu/?doc=https://hyro.tech)

I've never cared enough about it because it's not required in HTML5, but might as well make Modesta support HTML4 :P

Yes, but what I have read may happen that browsers will stop supporting HTML4, so I would like to ask if you would do this for HTML5 @AlexFlipnote

This means that the browser will automatically put it there as if DOCTYPE html. And there is a problem that then the fullscreen class does not work and it bugs.

I also experienced this bug.

This will be fixed in v3 release


Adding splunk support bundle powershell script

Description:
Gathering host information using Power-shell script for Windows.
Script will collect all the required information from host. It works same as bash script for Linux.
It will collect logs of service, metrics data, and system information and store it in a tar.gz formate at temporary directory.

@djani-splunk Let's try adding some steps in our circleci job (https://github.com/signalfx/splunk-otel-collector/blob/main/.circleci/config.yml#L333) to execute/verify the script after MSI installation.

@djani-splunk Let's try adding some steps in our circleci job (https://github.com/signalfx/splunk-otel-collector/blob/main/.circleci/config.yml#L333) to execute/verify the script after MSI installation.

Suggested changes have been done. Kindly review it and let us know if further changes are required.

@jcheng-splunk can you review?

HI @jcheng-splunk
Suggested changes are done and also tested. and now we can verify logs of td-agent.
You can verify changes and guide if more improvements are required.


Review the formula

Hi, could you review the formula, please:
PSA uses the following formulas to calculate the tracking metrics:

Percentage of cost consumed = Actual cost spent to date ÷ Planned cost for the task
Cost to complete (CTC) = Planned cost – Actual cost spent to date
EAC = CTC + Actual cost spent to date

Is it correct that "EAC" is same as "Planned cost"?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b4064178-0c68-edf5-12ff-e98f2a8d2804
Version Independent ID: cf6ff194-42fa-cda9-73c0-c2b3b41604e0
Content: Project progress and cost consumption
Content Source: ce/project-service/project-tracking.md
Service: dynamics-365-customerservice
GitHub Login: @ruhercul
Microsoft Alias: ruhercul

Sorry for the late reply.
Using Effort:
Effort Estimate at Completion = Remaining Hours + Actual Hours
We hold planned hours constant (only editable from the WBS) as a pseudo baseline.   The remaining hours are used to forecast what it will take to complete the task.
Regards,
Russel
From: MariaChikitkina<EMAIL_ADDRESS>Sent: Thursday, July 11, 2019 5:03 AM
To: MicrosoftDocs/dynamics-365-customer-engagement<EMAIL_ADDRESS>Cc: Russel Hercules<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: [MicrosoftDocs/dynamics-365-customer-engagement] Review the formula (#1191)
Hi, could you review the formula, please:
PSA uses the following formulas to calculate the tracking metrics:

Percentage of cost consumed = Actual cost spent to date ÷ Planned cost for the task
Cost to complete (CTC) = Planned cost – Actual cost spent to date
EAC = CTC + Actual cost spent to date

Is it correct that "EAC" is same as "Planned cost"?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b4064178-0c68-edf5-12ff-e98f2a8d2804
Version Independent ID: cf6ff194-42fa-cda9-73c0-c2b3b41604e0
Content: Project progress and cost consumptionhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fdynamics365%2Fcustomer-engagement%2Fproject-service%2Fproject-tracking%23feedback&data=02|01|russel.hercules%40microsoft.com|3cdbd8b4d7e148e6445b08d705f7b8c6|72f988bf86f141af91ab2d7cd011db47|1|0|636984433830508280&sdata=mZyX%2FSy3aj%2BOE6wwknDlpQZPvMRnXcKssYivYQELlFU%3D&reserved=0
Content Source: ce/project-service/project-tracking.mdhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fdynamics-365-customer-engagement%2Fblob%2Fmaster%2Fce%2Fproject-service%2Fproject-tracking.md&data=02|01|russel.hercules%40microsoft.com|3cdbd8b4d7e148e6445b08d705f7b8c6|72f988bf86f141af91ab2d7cd011db47|1|0|636984433830508280&sdata=FuPH%2BVhrHcs14wDtaVOjnjck0hkHMethIyVWl54JHjA%3D&reserved=0
Service: dynamics-365-customerservice
GitHub Login: @ruherculhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fruhercul&data=02|01|russel.hercules%40microsoft.com|3cdbd8b4d7e148e6445b08d705f7b8c6|72f988bf86f141af91ab2d7cd011db47|1|0|636984433830518273&sdata=zQuW2cS9%2FZ6otmmpVSWmS1kpyR91pe5k5rCqc24v1mA%3D&reserved=0
Microsoft Alias: ruhercul

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fdynamics-365-customer-engagement%2Fissues%2F1191%3Femail_source%3Dnotifications%26email_token%3DAJL5MVY5T4KQVK44WEKYXHTP64OPHA5CNFSM4IBCTZKKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4G6TSMFA&data=02|01|russel.hercules%40microsoft.com|3cdbd8b4d7e148e6445b08d705f7b8c6|72f988bf86f141af91ab2d7cd011db47|1|0|636984433830518273&sdata=ZQgzIF66RclRE29qvftcd3VZM15YsTHPbnOlaiAvW98%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAJL5MV5IKAIUVYJCAGDUXSTP64OPHANCNFSM4IBCTZKA&data=02|01|russel.hercules%40microsoft.com|3cdbd8b4d7e148e6445b08d705f7b8c6|72f988bf86f141af91ab2d7cd011db47|1|0|636984433830528268&sdata=P2wmBaCZMwTngd6Rxpfg3330Z3R7RHDrlR4AGWWjwNY%3D&reserved=0.


Scalability

As discussed at a recent Monday morning meeting, some of our Apps are expecting input files to  have one paragraph per line.  Each paragraph is treated as a separate document and processed.  This works well with the files that were converted from PDFs by hand, like the 8 or 10 we have been using for testing.  With the full set of 52 which were automatically converted, however, this doesn't work well.  The PDF converter seems to have output one line per PDF line and not attempted to make paragraphs.  These lines are usually not even complete sentences.  The way to handle this is to treat the entire file as a document and let the Eidos system extract the sentences.  This isn't working well.  Many files never run to completion.  I made a small and informal test program that uses a text of approximately 4KB from CAG.fulltext and am timing linear increases in that text length.  It doesn't look good.  A 16KB file is taking around 30 minutes and the largest of the 52 test files is still 10 times larger.

Thanks @kwalcock!
We've seen this behavior before: the text produced from the PDF reader has huge sentences because of lack of punctuation. Some of the components we have depend on sentence length, so they are completely off for very long sentences.
One simple solution I suggest: insert an artificial period whenever the text contains two new lines wo/ any text in between. This usually indicates a new paragraph, and we should treat it as such, even when no punctuation is recognized.
Also, I'd like to see an example of these files that take forever to parse.

We could have the problem you mention, but this test is being performed with known, fairly well behaved text.  It's pasted below.  To make the document longer, I just concatenate another copy.  We are probably doing more searching and sorting than previously and perhaps I should comment it out to see if that's the culprit.
 In Eastern Equatoria - one of the drought-hit areas - the latest food security and nutrition assessments in November 2016 revealed that households experiencing poor food consumption nearly doubled from 19 to 37 percent when compared with the same time in 2015. Rainfall deficits were widely reported in the area, with 93 percent of households indicating this was a major shock during the last agricultural season, impacting pasture and water availability and local food production. Climatic forecasting indicates these areas are likely to experience depressed rainfall between March to May 2017.
South Sudan's economic crisis has been driven by the rapidly depreciating value of the South Sudanese Pound (SSP), shortages of hard currency, global declines in oil prices and significant dependence on imports. Conflict, insecurity, market disruption, economic downturn and localized crop failures have caused record high food prices and hunger has spread to locations that were previously stable.  Conflict and economic decline have led to violence and displacement.  Violence has caused livestock to be looted, killed and disease-prone and crops destroyed, and displacement has caused late planting.  These impacts on livestock and crops have resulted in livelihoods being decimated.
Food insecurity is becoming more severe across a wider area and is deepening for people already made vulnerable by displacement and conflict. In 2017, food insecurity in Unity, Jonglei and parts of Greater Equatoria and Greater Bahr el Ghazal remained critical as spikes in conflict, economic collapse, and impacts of flooding reduced agricultural production. About three quarters of counties countrywide are expected to face severe food insecurity in the first quarter of 2018. Even after the harvest in late 2017, food prices remained high, and the 2018 lean season is projected to begin early and become worse than in 2017. Particularly for people living in market-dependent urban areas, economic decline has caused a reduction in access to staple food, clean water, and to a variety of foods.
The rising cost of living and impact of the conflict have also undermined people's ability to access safe water. It is estimated that only 13 per cent of South Sudanese people have access to improved sanitation, while 85 per cent of the population practice open defecation and only 41 per cent have access to safe water. Families in urban centres have had to spend an increasing portion of their income to obtain clean water, while water trucking has decreased due to the cost of fuel. Borehole repairs have not been possible in areas hardest hit by conflict, including large swathes of Upper Nile, due to lack of access due to insecurity and lack of technical expertise and supplies.
Due to conflict and food shortages, more than one in five people in South Sudan have been forced to flee their homes in the past 22 months, including over 1.66 million people who are currently internally displaced and nearly 646,000 people who have fled to neighbouring countries as refugees. Many have been displaced multiple times because of repeated attacks, particularly in counties such as Leer, Koch, Mayendit and Rubkona in Unity State, Fangak and Pigi County in Jonglei, and Malakal and surrounding areas in Upper Nile.
Persistent insecurity and armed conflict have disrupted livelihood activities, affected market functionality and shrunk physical access to markets. Acute malnutrition has worsened compared to the same period in 2016 due largely to the unprecedented high levels of food insecurity, widespread fighting, displacement causing poor access to services, high morbidity, extremely poor diet (in terms of both quality and quantity), low coverage of sanitation facilities, and poor hygiene practices. While marginal improvements in levels of acute malnutrition are expected up to December 2017 due to consumption of household production, forecasts for 2018 are deeply concerning with over 1.1 million children under five expected to be acutely malnourished and 269 000 children likely to be severely malnourished.

Yes, the sentence boundaries are reasonable here...
Can you please do an ablation test to understand where the most significant slowdown is? Is it parsing? In the grammars? In grounding? In the code following the grammars?
Thanks!

The program has been running with a profiler since yesterday and there will be results soon.
I wonder how the artificial period you suggested previously will affect provenance if it is inserted outside of the processor.  Could the situation be handled by what you are working on now with the tokens, some of which are being inserted behind the scenes like ", and"?  If some sort of "paragraph breaker" was in the pipeline, it could look for "not end of sentence puncutation"-newline-newline and create the artifical token to make sure the (potential) sentence has ended.  The forthcoming interval correction would take care of provenance considerations.

In the current design, but perhaps not the new one Becky is working on, the few lines of keepCAGRelevant were a hotspot.  This is because Mentions are repeatedly compared to each other there, and really a very slow Mention.equals is the culprit.  It uses the hashCode.  hashCode is a def for Mentions, meaning that it is always recalcuated.  A simple equals can bail at the first unequal field, but hashCode doesn't.  So, it is always recalculated and calculated to the end, making it very slow.  I added a local hashCode buffer and the execution time for keepCAGRelevant went from 170 seconds to 0.664 seconds.  I'm checking if anything else makes lots of comparisons.  This may be a very good place to use a lazy val if the Mention state doesn't change and it doesn't mess up subclasses.  That would mean a change to processors, though.

The next hot spot was State.mergeLuts which has a line mentions.distinct which will go repeatedly through all the mentions and check for equality.  Since this is already in Processors, I added caching to the Mention hashCode.  It looks promising so far.  Playing with object identity is dangerous, so several people should look at this in the end.  The longer text should at least finish.

Nice work, Keith!
On Tue, May 22, 2018 at 4:35 PM Keith Alcock<EMAIL_ADDRESS>wrote:

The next hot spot was State.mergeLuts which has a line mentions.distinct
which will go repeatedly through all the mentions and check for equality.
Since this is already in Processors, I added caching to the Mention
hashCode. It looks promising so far. Playing with object identity is
dangerous, so several people should look at this in the end. The longer
text should at least finish.
[image: image]
https://user-images.githubusercontent.com/8679738/40395649-fb49c052-5ddd-11e8-8462-c847c30921dc.png
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/clulab/eidos/issues/316#issuecomment-391173926, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFIniSPxpajouecGYWriCqkfKDkN_HV2ks5t1KCYgaJpZM4UD9G2
.

At a recent meeting I think that somebody suggested running through the PDF files that @EgoLaparra has collected.  I did a quick pdftotext over the ones I found on Google Drive and see that a 2016/White_Paper/v118rd550.txt is about 1100KB.  The guestimated processing time is around 3.6 hours.
Heads up.  So far we've only needed to turn in things that we've processed on our own time.  Will we ever need to do something like this while some judge waits for output?

Thanks! Can you explain what these plots mean?

Some lines are just old timings run after some different kinds of optimization were performed.  We should be on the grey curve now, but it is quite an extrapolation.  The x-axis units were ~4KB texts and I see that I even divided wrong, so here's a version with more standard units.  It looks like it might approach 18,000 seconds or 5 hours for the largest file.  I don't know about the memory requirements, though.  We can't go many times higher on that.  The napkin check on feasibility leads me to conclude that it will take a lot of patience to process all the files.

Thanks @kwalcock!
I wonder why the growth is super linear... You did find some issues in Odin code. Any other ideas?
(This is not urgent)

I wonder how some things like List.distinct are implemented.  It could easily be n^2.  The first tests didn't go far enough up the curve to show an obvious culprit.

The largest file in the 17.5k collection of files on Google Drive is 3740KB.  This isn't going to finish in a reasonable time.  There are numerous (>90) 1MB files that will take around 4 hours each.  That's not really reasonable, either.  What to do, @MihaiSurdeanu?

Good catch. In Reach, we simply skipped files > 1MB. I suggest the same strategy here. Maybe we can be even more aggressive on this threshold? Say 0.5MB?

This is not very accurate, especially on the low end, but it shows an estimate of processing time for each file and then total processing time if they are sorted from smallest to largest.  Based on this, it would take several thousand hours to process all 17k documents.  The smallest 20% of them might be done in around one hour.  40% of them might be processed overnight in 10 hours.  (I'm not sure what Excel is doing with the out of place orange dot.)

To what extent might this be a solution to indexing: don't include the indra data in advance?  Instead, we index all the text files and include the file names and file sizes in the index.  Someone does a search and the answer is the names and sizes of the files that would need to be processed in order to see something interesting.  Something downstream can filter out files that are too large or ones with low scores and run eidos or indra on whatever is though suitable.

I like your last point (dividing the files into chapters) for the files we have to process. But for the purpose of this exercise, I would:

Fix the bug that you discovered (assigned to @bsharpataz).
The rerun Indra on the smallest 40 - 50% of the files. Can you please generate this list, so we can send it to Patrick?

Just to be sure...  It's Eidos that I've been running.  I've been assuming that Patrick was taking the Eidos output and running Indra on it.

No, Indra calls Eidos programmatically. I don't know if they can read Eidos's output into Indra. Can you please ask Patrick if they have this functionality? This would save them some CPU cycles...

Sure, if Eidos already produced some output as JSON-LD, INDRA can just process those into Statements the reading isn't required to be done with the involvement of INDRA (we just do it that way because then we can control it through a Python API).

I've got a test running to check the feasibility of processing all files that are less than 20,000 bytes, which should be 40% of the collection.  It will produce a large file that might be difficult to transfer.  Can someone specify which ontologies and which vectors?

@bsharpataz ?

I'll go ahead and use all ontologies and glove.

That's what i would have said, but fwiw if we see diffs among ontologies,
you may want to check with MESH ontology which is far larger than any of
ours
On Tue, Aug 14, 2018 at 6:09 PM Keith Alcock<EMAIL_ADDRESS>wrote:

I'll go ahead and use all ontologies and glove.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/clulab/eidos/issues/316#issuecomment-413063998, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFIniYBGI74Bwv-uovcz-w5Mpq4zpD3kks5uQ3ThgaJpZM4UD9G2
.

It did better than expected and got through all text files with sizes less than 50,000 bytes, which is the shorter 60% of the files.  A zip file is being prepared.


How can I skip the ‘ polish‘’ and ‘split’ step and just cluster and order raw contigs ?

hello，
I am using 3DDNA to  perform contig scaffolding,
I just need to cluster and order my contig like 'Lachesis', and don't want to do any polish or split
Which parameter should I use ?
I try to use '-r 0'  parameter,
'$ 3d-dna/run-asm-pipeline.sh -r 0 --sort-output   references/draft.fasta   merged_nodups.txt  '
But still got polish.hic and split.hic and a final corrected scaffolds.（that is not my required）
I also tried to use '--early-exit' ,
'$  3d-dna/run-asm-pipeline.sh  --early-exit  references/draft.fasta   merged_nodups.txt  '
but did not get a final.fasta file
I want to skip the ‘ polish‘’ and ‘split’ step and just cluster and order raw contigs  .
colud U help  me?
@sanjitsbatra @dudcha @theaidenlab

Hello,
Please note that there is no separate clustering step in 3d-dna. If you want to get the first step only which is ordering and orienting the draft sequences you can use the early exit flag.
Olga

On Sep 21, 2020, at 9:51 PM, kunproj<EMAIL_ADDRESS>wrote:

hello，
I am using 3DDNA to perform contig scaffolding,
I just need to cluster and order my contig like 'Lachelis', and don't want to do any polish or split
Which parameter should I use ?
I try to use '-r 0' parameter,
'$ 3d-dna/run-asm-pipeline.sh -r 0 --sort-output references/draft.fasta merged_nodups.txt '
But still got polish.hic and split.hic and a final corrected scaffolds.（that is not my required）
I want to skip the ‘ polish‘’ and ‘split’ step and just cluster and order raw contigs .
colud U help me?
@sanjitsbatra @dudcha @theaidenlab
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.


sbt deploy/publish fails with java 12 enable preview feature | JavadocGenerationFailed

Hi Team,
JDK: 12
SBT: 1.3.4
Scala: 2.12.9
Play : 2.7
sbt debian:deploy or debian:publish command also fails with the below error
[warn] Constructing Javadoc information...
[warn] error: classfile for XXXXXXX/Indexable.class uses preview features of Java SE 12.
[warn]   (use --enable-preview to allow loading of classfiles which contain preview features)
[warn] error: classfile for XXXXXXX/Client.class uses preview features of Java SE 12.
[warn]   (use --enable-preview to allow loading of classfiles which contain preview features)
[warn] error: classfile for XXXXXXX/ConfigOptional.class uses preview features of Java SE 12.
[warn] 15 errors.
[warn] javadoc exited with exit code 1
model contains 324 documentable templates
[info] Main Scala API documentation successful.
[error] sbt.inc.Doc$JavadocGenerationFailed

but we pass "--enable-preview" in javacOptions & javaOptions & setting JAVA_OPTS="--enable-preview".
sbt.inc.Doc$JavadocGenerationFailed
[error] 	at sbt.inc.Doc$$anon$1.run(Doc.scala:72)
[error] 	at sbt.inc.Doc$$anon$2.run(Doc.scala:91)
[error] 	at sbt.inc.Doc$$anon$3.$anonfun$run$5(Doc.scala:118)
[error] 	at sbt.inc.Doc$$anon$3.$anonfun$run$5$adapted(Doc.scala:114)
[error] 	at sbt.util.Tracked$.$anonfun$inputChanged$1(Tracked.scala:150)
[error] 	at sbt.inc.Doc$$anon$3.run(Doc.scala:122)
[error] 	at sbt.Defaults$.$anonfun$docTaskSettings$3(Defaults.scala:1642)
[error] 	at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error] 	at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
[error] 	at sbt.std.Transform$$anon$4.work(Transform.scala:67)
[error] 	at sbt.Execute.$anonfun$submit$2(Execute.scala:281)
[error] 	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:19)
[error] 	at sbt.Execute.work(Execute.scala:290)
[error] 	at sbt.Execute.$anonfun$submit$1(Execute.scala:281)
[error] 	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
[error] 	at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[error] 	at java.base/java.lang.Thread.run(Thread.java:835)
[error] (search / Compile / doc) sbt.inc.Doc$JavadocGenerationFailed

Any suggestions?
Originally posted by @Santhoshas in https://github.com/sbt/sbt/issues/5021#issuecomment-849574985

Thanks for the report. Could you provide reproduction steps so we can reproduce your problem on our computers?

Hi Team,
We are trying to upgrade our play modules to JDK 12 with the below versions
JDK: 12
SBT: 1.3.4
Scala: 2.12.9
Play : 2.7
Below lines are added to build.sbt to avoid the below-mentioned issues.
Issue:
use --enable-preview to allow loading of classfiles which contain preview features
Lines:
+javacOptions ++= Seq("--enable-preview", "--release", "12")
+javaOptions ++= Seq("--enable-preview")
sbt clean/compile looks fine but when we execute sbt debian:publish on Jenkins/local, the build fails with the below-mentioned error.
Reason: I think "javac" gets the enable-preview parameter (compile step is fine) but the implicit javadoc (or scaladoc) in the deploy doesn't seem to get that argument.

Thanks for the report. Could you provide reproduction steps so we can reproduce your problem on our computers?

Do you need any other details?

I haven't tried to repro it, but hopefully it's enough info for someone to get started on it.
The relevant area is likely https://github.com/sbt/zinc/blob/develop/internal/zinc-compile-core/src/main/scala/sbt/internal/inc/javac/LocalJava.scala.

I haven't tried to repro it, but hopefully it's enough info for someone to get started on it.
The relevant area is likely https://github.com/sbt/zinc/blob/develop/internal/zinc-compile-core/src/main/scala/sbt/internal/inc/javac/LocalJava.scala.

Is it a bug from your end or something that needs to change on the application side? Can you please give some idea so that will proceed with a higher version?

Compile / doc / javacOptions ++= Seq("--enable-preview",  "--release", "12") solves the problem


Only 1 bot gets paused when running start.sh on first boot of the day

Even after changing user_count < 3 on first boot of the start.sh script only bot1 gets paused, when using 2 bots to connect to the same channel.
If I enter and exit the channel with a user they both pause as they should.
EDIT: After some testing it seems like the bot is not counting itself when starting the start.sh script and once the second bot joins it then counts the user state change but only auto pauses the first bot that entered the channel.
Possible to add or move parts of the auto start script to a def connected rather than def userstate?

The bot only react on user-state changes. But it react on ANY user-state changes. That means if the first user changes a room or join or left the server it will react with muting and pausing itself (If conditions for muting given).
If you want your bots not to play when the join the server it's the best to pause mpd in start-script.
If this is not your wanted behavior of the bot please describe HERE on witch condidion should happen what.
I'll read it then if I'm not burned out after a long day and try to implement

I will try and explain a bit better.
I have 2 bots that connect everyday at 9:00 am after a server reboot the previous night.
The start.sh script will connect 1 bot then the other, bot 1 registers that a userstate change has occurred when bot 2 enters and mutes and pauses itself. Bot 2 never mutes or pauses unless I enter the channel with another user then everything works as intended.
I guess your comment about having the bots start up paused would fix this issue.  Just wasn't sure if there was a way to have bot 2 also capture that it should also mute/pause itself by adding the script in a def connected rather than def userstate change.

That is true. It is generally no good idea to set a bot per default into a self unmuting and playing status.
When a user logged of the previous day in a channel where the bot was and return on the next day.... the first he was hear is .......
Maybe a other user has turnd up the volume to max.
--> heart attack :broken_heart: :smiling_imp:


improve error message for disallowed model[[nodes]] and model[[nodes[i]]] usage

Currently, we (unhelpfully) output:
> cmcmc <- compileNimble(mcmc, project = model, showCompilerOutput = TRUE) Compiling
    [Note] This may take a minute.
    [Note] On some systems there may be some compiler warnings that can be safely ignored.
Error in exists(as.character(varOrNodeExpr), x, inherits = FALSE) :
      first argument has length > 1

User message from 2022-12-07 has a reprex, but it would be easier to just make a minimal reprex when we pursue this.

Actually model[[nodes]] is well-trapped: Error: Length of nodes requested from model using '[[' is 2. It must be 1.

I've added error trapping for model[[nodes[i]] in determineNdimsFromNfproc. See PR #1259.


samples: basic: hash_map: libraries.hash_map.newlib.cxx_unordered_map.djb2 fails

Describe the bug
The samples/basic/hash_map/libraries.hash_map.newlib.cxx_unordered_map.djb2 fails
Observed for:

nrf5340dk_nrf5340_cpunet
nrf9160dk_nrf9160_ns
nrf9160dk_nrf9160

To Reproduce
Steps to reproduce the behavior:

have nrf9160dk_nrf9160 connected
go to your zephyr dir
call ./scripts/twister -T samples/basic/hash_map -p nrf9160dk_nrf9160--device-testing --device-serial /dev/ttyACM0 -v --inline-logs
See console output with error

Expected behavior
Valid console output
Impact
Not clear
Logs and console output
[2023-02-26T21:04:05.371Z] ERROR   -  CONFIG_TEST_LIB_HASH_MAP_MAX_ENTRIES: 40
[2023-02-26T21:04:05.371Z] [00:00:01.766,143] <err> os: ***** USAGE FAULT *****
[2023-02-26T21:04:05.371Z] [00:00:01.766,143] <err> os:   Stack overflow (context area not valid)
[2023-02-26T21:04:05.371Z] [00:00:01.766,174] <err> os: r0/a1:  0x20001cb8  r1/a2:  0x00000000  r2/a3:  0x00004070
[2023-02-26T21:04:05.371Z] [00:00:01.766,174] <err> os: r3/a4:  0x00000000 r12/ip:  0x20001ef8 r14/lr:  0x000020ed
[2023-02-26T21:04:05.371Z] [00:00:01.766,174] <err> os:  xpsr:  0x49000000
[2023-02-26T21:04:05.371Z] [00:00:01.766,204] <err> os: Faulting instruction address (r15/pc): 0x000014f6
[2023-02-26T21:04:05.371Z] [00:00:01.766,235] <err> os: >>> ZEPHYR FATAL ERROR 2: Stack overflow on CPU 0
[2023-02-26T21:04:05.371Z] [00:00:01.766,265] <err> os: Current thread: 0x20000d60 (unknown)
[2023-02-26T21:04:05.371Z] [00:00:01.828,125] <err> os: Halting system

Environment (please complete the following information):

OS: Ubuntu 20.04.1 LTS
Toolchain Zephyr SDK 0.15.2
Commit SHA or Version used:  zephyr-v3.3.0-472-gb2512d2f534e

found similar issue in https://github.com/zephyrproject-rtos/zephyr/issues/55166

@nordic-piks - are you able to suggest a min_ram setting or minimal stack size that does make the example work on your board (or at least be filtered out due to ram constraints)?
Not able to reproduce currently because I cannot flash my nRF device from macOS (text below is from discord).

Does anyone know if it's possible to flash the nrf52840dongle_nrf52840 using macOS? I tried using west build -p auto -b nrf52840dongle_nrf52840 -t flash sample/hello_world but I get the error below. Just trying to repro / fix a stack overflow on nrf HW.
-- west flash: using runner nrfjprog
FATAL ERROR: "nrfjprog --ids" did not find a board; is the board connected?

But I actually do have nrfjprog installed:
nrfjprog -v
nrfjprog version: 10.17.2 external
JLinkARM.dll version: 7.70a

It seems to just not work at all on macOS (yes, dongle is plugged-in)
% nrfjprog --ids
%

@PerMac @anangl Can you help with those 2 issues? (stack size and nrfjprog issue).
@cfriedt What device you have?

@PerMac @anangl Can you help with those 2 issues? (stack size and nrfjprog issue).
@cfriedt What device you have?

I ended up downloading the prebuilt mac release from Nordic and was able to get things almost working then, but had to switch tasks.
I do have the 52840 dongle and devkit. I think nrfprog should work with the latter, right?

Yes, using devkit with be the best option - should work out of the box.

@cfriedt I think there was some misunderstanding. The test works on nrf52840dk, you won't be able to reproduce the failure with it. Failures were observed for:

nrf5340dk_nrf5340_cpunet
nrf9160dk_nrf9160_ns
-nrf9160dk_nrf9160
Not sure about nrf52840 dongle, we don't have it in our CI, nor on my desk. I can ask my colleagues if they can try this on 52 dongle.

Ah, I see. That's all I have for Nordic parts locally

It seems you want be able to reproduce it with dongle either. From what I got it doesn't have programator on it, so without soldering something there you won't be able to use nrfjprog to program. Dfu/mcu is needed. I will try to help figuring out is some stack can be increased to simple solve the issue.

It's possible that simply setting min_ram for the cxx test permutation might solve the issue. Is C++ STL used a great deal on these parts?

I played with stack sizes and found that adding
CONFIG_MAIN_STACK_SIZE=2048
to prj.conf solved the issue on all the platforms we observed it. Setting 1024 was not yet enough. Not sure will it be efficient on stm platforms
@FRASTM Is setting such stack size solves your issues? When debugging I found that nrf5340_cpunet was also either passing or printing nothing

I played with stack sizes and found that adding CONFIG_MAIN_STACK_SIZE=2048 to prj.conf solved the issue on all the platforms we observed it. Setting 1024 was not yet enough. Not sure will it be efficient on stm platforms @FRASTM Is setting such stack size solves your issues? When debugging I found that nrf5340_cpunet was also either passing or printing nothing

Yes, I noticed the STACK SIZE was too low (https://github.com/zephyrproject-rtos/zephyr/issues/55166), but I did not find the lower limit.

I played with stack sizes and found that adding CONFIG_MAIN_STACK_SIZE=2048 to prj.conf solved the issue on all the platforms we observed it. Setting 1024 was not yet enough. Not sure will it be efficient on stm platforms @FRASTM Is setting such stack size solving your issues? When debugging I found that nrf5340_cpunet was also either passing or printing nothing

@PerMac - if you are ok with it, please make a PR to add extra_configs the testcase.yaml for the cxx test case. If not, I can, but I'm not able to reproduce on any platforms atm.

see https://github.com/zephyrproject-rtos/zephyr/pull/55349


Marriage phase one

added first two slide decks: one created "by hand" and another one created by marpCombiner.
See sidebar on left

This includes all changes to previous PR. Sorry for that.


Ipc add countrystatus

This adds some info level logging of how countries were processed in food security

Pull Request Test Coverage Report for Build<PHONE_NUMBER>2
Details

10 of 10 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.04%) to 93.11%

Totals

Change from base Build<PHONE_NUMBER>4:
0.04%

Covered Lines:
1527

Relevant Lines:
1640

💛  - Coveralls


add IPAddrFromNetIP function to allow convertion from NetIP to IPAddr

Codecov Report

Merging #41 (fdec2f2) into main (80443b0) will increase coverage by 0.20%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main      #41      +/-   ##
==========================================
+ Coverage   46.89%   47.10%   +0.20%
==========================================
  Files          16       16
  Lines        1032     1036       +4
==========================================
+ Hits          484      488       +4
  Misses        524      524
  Partials       24       24

Flag
Coverage Δ

unittests
47.10% <100.00%> (+0.20%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/flow/record.go
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 80443b0...fdec2f2. Read the comment docs.


Needed to set NO_SONAME for Linux to link.

I have tested and use this find module on Linux platform on a daily basis. I assure you this change is not necessary.
To use TBB in your application on linux, you need to install the dev package (on Ubuntu it is libtbb-dev), which includes the header files and creates a symbolic link for the library to a .so file name. Also, the header files are not installed unless you have installed the dev package, which would also cause this find module to fail.
There maybe use cases where TBB is needed as a dependency only, in which case the dev package would not be strictly necessary. However, for such a use case, FindTBB.cmake would need to be modified so that it did not fail when the header files are not found. Unfortunately such a change would make it impossible to check the version of TBB.

To use TBB in your application on linux, you need to install the dev package (on Ubuntu it is libtbb-dev), which includes the header files and creates a symbolic link for the library to a .so file name.

We do not use tbb from a package but the binary distribution. And then FindTBB fails.

We do not use tbb from a package but the binary distribution. And then FindTBB fails.

Are you referring to the TBB binary packages from the TBB web page?

Are you referring to the TBB binary packages from the TBB web page?

Yes exactly.

Ouch. Just noticed that just this change alone does not make it work.
I think I will not rather try to build TBB with CMake. Seems to be the easier route.

I am looking at the package right now and it definitely includes the .so files. I suspect that CMake is selecting the wrong library file for some reason. Perhaps due to a configuration variable you have set in your project. If you can provide a minimal failing example and you system details (compiler, os, and version numbers), I would be happy to help you resolve the issue.

One thing you can try is to run the test.sh script included in this project. It will allow you to verify that the installed TBB is working. I verified the find module against the binary package a while ago, but things can change.

Thanks for your effort.
Due to Wenzels great work I now have TBB statically linked on Linux. Will probably switch the other platforms, too.


fixes: pid reuse attack when kill a exec process

Because of we use unix.Kill to kill a exec process, but after a exec process has been stopped, the pid in execProcess has not been deleted, we also can use execProcess.Kill to kill the process with the same pid.
use unix.Kill
https://github.com/lifubang/containerd/blob/831a41b9585fc021c1036da17afa1513e7b4f908/runtime/v1/linux/proc/exec.go#L145-L153
don't delete pid in execProcess
https://github.com/lifubang/containerd/blob/831a41b9585fc021c1036da17afa1513e7b4f908/runtime/v1/linux/proc/exec.go#L96-L101
recall execProcess.Kill
https://github.com/lifubang/containerd/blob/831a41b9585fc021c1036da17afa1513e7b4f908/runtime/v1/linux/proc/exec_state.go#L153-L155
So, if the pid is reused, the wrong process will be killed.
Signed-off-by: Lifubang<EMAIL_ADDRESS>

Codecov Report

Merging #2832 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #2832   +/-   ##
=======================================
  Coverage   43.79%   43.79%
=======================================
  Files         100      100
  Lines       10741    10741
=======================================
  Hits         4704     4704
  Misses       5307     5307
  Partials      730      730

Flag
Coverage Δ

#linux
47.45% <ø> (ø)
:arrow_up:

#windows
40.96% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 75c490c...f31c097. Read the comment docs.

I see you opened a backport for the release/1.1 branch; I guess this is also needed for the 1.2 branch (and possibly 1.0 branch?) (looking at https://github.com/containerd/containerd/blob/master/RELEASES.md#support-horizon, 1.0 is still supported)

Do we really need to consider this case? Who are we trying to protect from?

I think this can protect some one who just use lib containerd in their program. And may protect moby when the event queue is blocked or slow for some reasons, because docker stop will call p.Kill first.

I think the pid reuse attack may happen needs a long time, in a short time it have very very small probability. If we want to fix this situation, we can use ProcessStartTime check like runc.

I see you opened a backport for the release/1.1 branch; I guess this is also needed for the 1.2 branch (and possibly 1.0 branch?) (looking at https://github.com/containerd/containerd/blob/master/RELEASES.md#support-horizon, 1.0 is still supported)

Thanks, I'll open PR for these versions if this patch is taken by maintainers.

@Random-Liu in hear https://github.com/containerd/containerd/blob/96eb3bfa5dda628678ec2edc6d6c0b99e0fb119d/runtime/v1/linux/proc/exec.go#L254-L257 , if the pid is reused, the process can't delete anymore.


daemonset for logging is created in pod network

Hi,
while deploying logging archive only, the daemonset is created in Pod network instead with cluster node IPs.
Is there a paramater to adjust this deployment accordingly with helm?

Hi Marco!
Can you tell us more about what behavior you are expecting, or more on what the challenge you face is?
The logging pods don't deploy with any endpoints or services, as they only need to send data out of the cluster, are you trying to reach them?

Hi
issue got resolved locally


#85 Add configuration for ContentType

This change allows the user to configure the message content-type with an optional parameter or using the PSSlackConfig as requested in the linked issue.

Oy, great idea, and completely missed this PR!
Will check this out in the next day or two.
I... don't know enough to answer this, but will switching to use utf-8 by default potentially break anything for folks out there?  I might tweak this to initialize that config value as $null, and only set the parameter on invoke-restmethod if the config (or function param) specifies.  It's late though and I may not be thinking clearly : )


[✨] Remove this warning: routeLoader$() can only be declared in layout.tsx, index.tsx and plugin.tsx inside the src/routes directory

Is your feature request related to a problem?
Yes,
we want to reuse some routeLoader$ functions across multiple routes, and in many projects.
This means that we create a file called Loader.jsx and have a routeLoader$ in it. Yet it's not used directly.
It exports the function, and then we import it in another place:
For example, this is our BlogLoader.jsx file:
import { routeLoader$ } from<EMAIL_ADDRESS>import {
    getFromCacheOrApi,
    useAsync
} from 'Base'
import { getLayout } from 'Contents'
import { getForm } from 'Forms'
import { getGlobalization } from 'Globalization'

const loadPost = routeLoader$(async (props) => {
    const {
        params,
        status,
        query
    } = props
    const { slug } = params || {}
    const locale = query?.get('locale')
    const [
        data,
        layout,
        globalization,
        form
    ] = await useAsync([
        getFromCacheOrApi(`/blogPost/get?slug=${slug}&locale=${locale}`),
        getLayout('post', props),
        getGlobalization(props),
        getForm(`comment`, props)
    ])
    if (data.error?.code === "404") {
        return status(404)
    }
    if (data.statusCode) {
        return status(data.statusCode)
    }
    const { content, ...contentLessForm } = form
    return {
        ...data,
        ...layout,
        ...globalization,
        ...contentLessForm
    }
})

export default loadPost

This loader can be used in blog/index.jsx or artciles/index.js or news/index.jsx. For example, this is our blog/index.jsx in our Blog module:
import { component$ } from<EMAIL_ADDRESS>import { NotFound } from 'Base'
import { useSeo } from 'Seo'
import {
    Layout,
    loadBlog,
} from 'Blog'
import { loadBlog as runnableLoader } from 'Loaders'
import { Layout as RunnableLayout } from 'BlogParts'

const Blog = component$(() => {

    const data = loadBlog().value
    const extraData = runnableLoader().value
    if (data === 404) {
        return <NotFound />
    }

    const {
        categories,
        latest,
        mostViewed,
        popular,
        posts,
        seo,
        tags,
    } = data || {}

    return RunnableLayout
        ?
        <RunnableLayout {...data} />
        :
        <Layout {...data} />
})

export default Blog

export { loadBlog }

export { runnableLoader }

const head = ({ resolveValue }) => {
    return useSeo(loadBlog, resolveValue)
}

export { head }

It works just great. It utilizes the very fundamental import/export capability of JS and yet when we develop our project or build it and deploy it we see tons of warnings complaining that:

vite-plugin-qwik: routeLoader$() can only be declared in layout.tsx, index.tsx and plugin.tsx inside the src/routes directory, instead it was declared in "/Holism/SiteQwik/src/Modules/Products/Loaders/LoadProducts.jsx".
Please check the docs: https://qwik.builder.io/docs/route-loader/

Describe the solution you'd like
Please remove this warning. You are being very very opinionated here. Let teams decide about the framework and architecture for themselves.
Describe alternatives you've considered
We could not find an alternative. We just suffer by tons of warnings, while we do valid JS. And Qwik is just a framework on JS, thus it should accept JS-valid stuff.
Additional context
No response

I think this warning is here to warn the developer that route loaders have to be called within a route and cannot be used at the component level (for waterfall reasons if my understanding is correct).
Maybe instead of

routeLoader$() can only be declared in layout.tsx, index.tsx and plugin.tsx inside the src/routes directory, instead it was declared in "/Holism/SiteQwik/src/Modules/Products/Loaders/LoadProducts.jsx".

We could implement an eslint rule that fires when a loader is being called in a component:

routeLoader$() hooks can only be called in layout.tsx, index.tsx and plugin.tsx inside the src/routes directory, instead it was called in "/Holism/SiteQwik/src/Modules/Products/Loaders/LoadProducts.jsx".

In the meantime, you can easily disable this warning in your eslintrc file with the following rule:
    "qwik/loader-location": "off",

I added that rule to the .eslintrc.cjs and it did not work. This is the content of that file:
module.exports = {
  root: true,
  env: {
    browser: true,
    es2021: true,
    node: true,
  },
  extends: [
    'eslint:recommended',
    'plugin:@typescript-eslint/recommended',
    'plugin:qwik/recommended',
  ],
  parser: '@typescript-eslint/parser',
  parserOptions: {
    tsconfigRootDir: __dirname,
    project: ['./tsconfig.json'],
    ecmaVersion: 2021,
    sourceType: 'module',
    ecmaFeatures: {
      jsx: true,
    },
  },
  plugins: ['@typescript-eslint'],
  rules: {
    '@typescript-eslint/no-explicit-any': 'off',
    '@typescript-eslint/explicit-module-boundary-types': 'off',
    '@typescript-eslint/no-inferrable-types': 'off',
    '@typescript-eslint/no-non-null-assertion': 'off',
    '@typescript-eslint/no-empty-interface': 'off',
    '@typescript-eslint/no-namespace': 'off',
    '@typescript-eslint/no-empty-function': 'off',
    '@typescript-eslint/no-this-alias': 'off',
    '@typescript-eslint/ban-types': 'off',
    '@typescript-eslint/ban-ts-comment': 'off',
    'prefer-spread': 'off',
    'no-case-declarations': 'off',
    'no-console': 'off',
    '@typescript-eslint/no-unused-vars': ['error'],
    '@typescript-eslint/consistent-type-imports': 'warn',
    'qwik/loader-location': 'off',
  },
};

I added it to rules node, last line.

Tested this one more time on another repo and it always works on my end.
Looking further into this, this warning definitely comes from the eslint rule as it is the only place where you can find it in the Qwik codebase.
Do you still have the warning in development? You should have it printed out in your IDE even when using jsx.
You could try removing "plugin:qwik/recommended" if you can't see eslint warnings (just to see if your errors disappear).

@maiieul even after removing the plugin:qwik/recommended we get those warnings. We're developing inside a docker container, thus we don't see or use eslint during the development in our VS Codes. We only see those warning when we first setup our projects, and then when we want to build it.

Since the warning comes from the eslint plugin, it means the eslint config is being used when you setup your projects and at build time. You probably need to find a way to disable it in there as well.

Another option could be to move your reusable loaders out of your source directory. The eslint config is applied based on the tsconfig on new starter projects.

My suggestion is to use eslint comment to disable the warning if you know what you are doing.

@mhevery, We added rules to eslint, but it had no effect. How should we disable it? What code should we add to eslint? Also, my feedback is that it's too opinionated not to let a loader be defined out of layout or index and be reused across many files. It's normal JS. Please reconsider this decision.

@Nefcanto you need to add a comment on top of the file where this warning gets generated.
/* eslint-disable qwik/loader-location */

This issue is related to #5555 .

@maiieul, that's highly unproductive. Do you mean we have to add that repetitive file everywhere? That's very bad.

@Nefcanto I know. This is a workaround for now.

@maiieul, thank you, but because we have many, many files in our modules, we literally can't ask our developers to put that in every file they have. May I ask what prevents us from configuring eslint centrally?

You can define "qwik/loader-location": "off" in your .eslintrc config

@gioboa as I have mentioned above, I added it, and it did not work. I'll try again.

I tried with a fresh 1.3.0 project too and it's working 🤷🏼♂

@gioboa this is an actual repository of our current code. It's not an MRE, yet testing it is not difficult.

git clone https://github.com/Nefcanto/QwikRouteLoaderEsLint
cd QwikRouteLoaderEsLint
npm install --legacy-peer-deps
npm run build

You should see tons of them.


Migrated KotlinSynthetics to ViewBinding

Purpose / Description
Migration to ViewBindings
Fixes
Fixes a part of #4
How Has This Been Tested?
Tested on Pixel 4a API 30, back navigation in all four fragments work as expected
Learning (optional, can help others)
https://developer.android.com/topic/libraries/view-binding/migration
Checklist
Please, go through these checks before submitting the PR.

[x] You have a descriptive commit message with a short title (first line, max 50 chars).
[x] You have commented your code, particularly in hard-to-understand areas
[x] You have performed a self-review of your own code
[ ] UI changes: include screenshots of all affected screens (in particular showing any new or changed strings)
[ ] UI Changes: You have tested your change using the Google Accessibility Scanner

I think we have KotlineSynthetics in other Fragments as well :(

Yeah I was thinking of doing it a bit by bit. It would be easy to review rather than ten-twelve classes all at once. More chances of getting something wrong unnoticed.

And this is why I wanted the #3 to be merged first 🥹

Yeah I was thinking of doing it a bit by bit. It would be easy to review rather than ten-twelve classes all at once. More chances of getting something wrong unnoticed.

Sure, if you think there are not many changes, you can do it at once as well. As a sanity check after we have fully migrated, remove the dependency for KotlinSynthetics

making commit fragment by fragment, can be merged all at once.

@anuragshukla06 since I din't had tasks for all scenario, haven been able to test all of the fragments. Once tested, will mark it ready. Meanwhile if you want to review each commit could parallely do as well.

All the commits are independent, so if one commit it changed it won't cause any conflict to other ones. So that's a good part.

@anuragshukla06 these five commits are checked, can be merged.

These three also done

@anuragshukla06 finally done with the migration, removed the dependency as well. Next on list is gradle upgrade.


Enable Service to support the Audio module

Description
As a step towards Muzikie/streamer#5, Servive can not index transactions of the Audio module by default. We should add required code base to the Gateway and Indexer micro services to support this module.
Motivation
Enable Service to support Muzikie modules
issues

#20
#22
#28
#30
#32
#26

All the issues are now closed and the epic is completed.


Add more pages

These pages are close. The one thing left to do is to fix the layout so that it doesn't take the entire width. But that looks like it's an issue on other pages like /events/2021-summit-spring and should probably be fixed in the common layout separate from this PR

FYI @HAUDRAUFHAUN, I think after this the main thing left to do is the recipes page

hey gang I've added in a placeholder recipes page and fixed up the design of a few of the pages!
here's a quick video with the changes i did
https://www.loom.com/share/c633a5dda5f94e6799e9e3eef77e1b40

so do i merge? also what is the gameplan for deploying this thing? do we check in with @kevmodrome ?

I think it'd be fine to merge soon. There's still more work to do, but probably easier for everyone to keep in sync with the changes checked in. I just tried running this locally and did notice one regression from your changes though. The top menu seems to be a bit squished. The events page also doesn't seem to be quite as well aligned as it was before

Hopefully we can fix up the recipes page before deploying. That one was a bit trickier. I think it was using some Routify helper that we need to recreate

This might be a problem with bigger screens. I could try fixing it in another pr if you want... :)

I added the recipes in a halfway working state. I added a TODO in the two pages below:

src/lib/components/recipes/CategoryTree.svelte
src/routes/recipes/index.svelte

If anyone would like to take it over and get it over the finish line I won't object :smile:  I think it could be helpful to merge this PR eventhough it's not quite done. It'd probably make it easier to collaborate and send additional PRs against the current state of things

@ben i have fixed the css for the events page and made some tweaks to the recipes page to make it a little nicer

screenshots

Cool. That looks much nicer! I finished populating the recipes page, so I'm afraid it looks a bit off again with the new rest of the data if you want to take another look at the CSS
There's one more thing left to do, which is that the table of contents on the recipes pages are missing. It looks to me like that's contained in src/lib/layouts/Recipe.svelte and src/lib/layouts/RecipeCategory.svelte and is maybe set in the mdsvex config in svelte.config.js. I've never used mdsvex before, so I'm not quite sure how that gets wired up

put some circles on the icons and squared them up.. a little trick i learned from steve schoger

Nice! Much better :smile:
I just pushed the recipe table of contents. Some of the recipes are showing up with horizontal scrollbars. Also, the icons are missing from the table of contents because they're trying to load as img instead of svg, but that's also broken on the current site, so shouldn't be a blocker

im fine merging

I'm fine merging as well

great work team!!


Private link 4 monitor

Latest improvements:
MSFT employees can try out our new experience at OpenAPI Hub  - one location for using our validation tools and finding your workflow.

Contribution checklist:

[x] I have reviewed the documentation for the workflow.
[ ] Validation tools were run on swagger spec(s) and have all been fixed in this PR.
[ ] The OpenAPI Hub was used for checking validation status and next steps.

ARM API Review Checklist

[ ] Service team MUST add the "WaitForARMFeedback" label if the management plane API changes fall into one of the below categories.
adding/removing APIs.
adding/removing properties.
adding/removing API-version.
adding a new service in Azure.

Failure to comply may result in delays for manifest application. Note this does not apply to data plane APIs.

[ ] If you are blocked on ARM review and want to get the PR merged urgently, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.
Please follow the link to find more details on API review process.

/azp run automation - sdk

Can one of the admins verify this patch?

Hi @arolshan ,
One question about the tag "2020-03" I added, since it's not the default tag, are there anyway to reference this specific tag while generating?
Or the question could be, can I change the default tag to "2020-03"?

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

Hi wouldn't it be easier to just search & replace to common types ?

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

I added some comments. Please take a look.

/azp run automation - sdk

/azp run automation - sdk

I added a comment about the error contract. Please take a look.

/azp run automation - sdk

@VeryEarly
There is lots of rename etc. that causing breaking change. I know you can do this on preview. But would you care to have a quick check on the BreakingChange CI and let us know whether all of these are intended.
I will need to pass it to management for merge, since CI fail on BreakingChange.

/azp run automation - sdk

/azp run automation - sdk

@VeryEarly
There is lots of rename etc. that causing breaking change. I know you can do this on preview. But would you care to have a quick check on the BreakingChange CI and let us know whether all of these are intended.
I will need to pass it to management for merge, since CI fail on BreakingChange.

Hi @weidongxu-microsoft ,
These rename are for unblock SDK generation fail for dotnet due to inconsistent naming convention.
These breaking change looks good except
"The new version has a different 'pattern' value than the previous one."
Sorry I'm not familiar with swagger spec rules but seems like I didn't change such property "pattern", can you help me understand what is this line for？

/azp run automation - sdk

/azp run automation - sdk

@VeryEarly
It appears to be caused by repalcing "../../../../../common-types/resource-management/v1/types.json#/parameters/ResourceGroupNameParameter" with "#/parameters/SubscriptionIdParameter".
The one in common-types got "pattern": "^[-\\w\\._\\(\\)]+$", defined.
https://dev.azure.com/azure-sdk/public/_build/results?buildId=344704&view=logs&j=08b5038d-1e84-5877-88da-859df8312b69&t=c1978ecd-fc8a-5291-6d7a-7dd459d8b50f&l=439

@VeryEarly
It appears to be caused by replacing "../../../../../common-types/resource-management/v1/types.json#/parameters/ResourceGroupNameParameter" with "#/parameters/SubscriptionIdParameter".
The one in common-types got "pattern": "^[-\\w\\._\\(\\)]+$", defined.
https://dev.azure.com/azure-sdk/public/_build/results?buildId=344704&view=logs&j=08b5038d-1e84-5877-88da-859df8312b69&t=c1978ecd-fc8a-5291-6d7a-7dd459d8b50f&l=439

Got it, so the reason to replace "../../../../../common-types/resource-management/v1/types.json#/parameters/ResourceGroupNameParameter" with "#/parameters/SubscriptionIdParameter" is because other specs need to be included in the same tag, they have "#/parameters/SubscriptionIdParameter" defined and used already which conflict with "../../../../../common-types/resource-management/v1/types.json#/parameters/ResourceGroupNameParameter".
These breaking changes are intended, thanks.

@VeryEarly
Got it. Please let @akning-ms know when you want to merge to master.


drop old desktop-only resource API for the new multiplatform resource API

Credits to @m-sasha for the help

Thanks for the PR, but we'll update the tutorial ourselves when the new resources API is considered stable.


How to fine-tune model for Relation Classification  to be used in the transformers library?

Hi,
I tried to fine-tune model for Relation Classification and use it for inference in the transformers library.
I created model folder by combining fine-tuned model binary file and + other config files required by the transformers library.
It crashes, most likely due to the incorrect entity_vocal_size param:

When I fine tune it using this repo, entity_vocab_size is set to 3
When I use LukeTokenizer it generates entity_ids tensor set to [2,3] which exceeds entity embedding dimension
Strangely finetuned model available from huggingface https://huggingface.co/studio-ousia/luke-large-finetuned-tacred/blob/main/config.json has entity_vocab_size set to 500000
Unfortunately I cannot set this value (500000) when fine-tuning since it crashes fine-tuning process

Could you provide instruction or hints on the entity_vocab_size param or general process?
Thanks,
Grzeorz

Hi @gchlodzinski,
We do not currently provide fine-tuning code that is based on the model implementation in the transformers library. The checkpoint file available in the huggingface's model hub is converted using this script with the files contained in the model file and fine-tuned checkpoint file.

Hi @ikuyamada,
Thank you so much for providing the script. It worked well and I was able to convert my fine-tuned model.
I used my custom dataset during fine tuning, currently supporting only 3 types of relations.
And besides obvious modifications to the script I had to comment out the following assertions in my case
assert len(missing_keys) == 1 and missing_keys[0] == "luke.embeddings.position_ids"
assert not unexpected_keys

since missing_keys in case of my model was empty. But still converted model worked well in the transformers library.

Hello,
I'm also trying to run this script to use a fine-tuned model with the Transformers library but I got the following error:
File "convert_tacred_checkpoint.py", line 95, in convert_luke_checkpoint
    missing_keys, unexpected_keys = model.load_state_dict(new_state_dict, strict=False)
  File "C:\Users\admin\anaconda3\envs\luke\lib\site-packages\torch\nn\modules\module.py", line 1224, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for LukeForEntityPairClassification:
        size mismatch for classifier.weight: copying a param with shape torch.Size([33, 1536]) from checkpoint, the shape in current model is torch.Size([32, 1536])

My custom dataset is composed of 32 relations. Removing the following lines ignores the error:
missing_keys, unexpected_keys = model.load_state_dict(new_state_dict, strict=False)
assert len(missing_keys) == 1 and missing_keys[0] == "luke.embeddings.position_ids"
assert not unexpected_keys

However, the generated model doesn't work as expected, it only predicts the same relation for any given entity pair...
@gchlodzinski, did you face a similar problem?

@fillipefbr  It looks like your dataset actually contains 33 relations instead of 32. Please confirm that label_list, created at the following line, consists only of 32 relations in your dataset.
https://github.com/studio-ousia/luke/blob/master/examples/relation_classification/main.py#L183

@fillipefbr I looked at my modified code and indeed I have commented out asserts so most likely I had similar problems.  And what @ikuyamada wrote might help to resolve the issue (in a better way than commenting out).
But despite commenting assertions out my models trained on a custom dataset worked correctly. I trained LUKE several times on different datasets, converted  it to transformers format and it was working with expected performance.

@ikuyamada, @gchlodzinski thank you for your feedback!
It turns out this line was adding a no_relation label to the dataset, which doesn't exist in my application. Modifying the generated label_list (as pointed by @ikuyamada) and retraining the model solved the issue. Now eveything works as expected!


Update all Go deps and apply minor fixes

Update all Go deps to the latest (much much better error messages now when trying to understand anchors and aliases with merge etc)
Fix minor breakage after update
Add some cmd help
Bump version so it's obvious the original isn't being used.

TODO - upstream this later when more of the TODOs have been done.


Support all Emoji's for Unicode 9

8Vim should be able to support all emoji's as specified in Unicode specification 9.

Done.


Can't interact with Appbar if GameObject with ObjectManipulator and BoundingBox has a Rigidbody

Describe the bug
If I create a Gameobject with the ObjectManipulator script and a Rigidbody attached, I can't use the AppBar prefab for that object, it doesn't detect inputs.
To reproduce
Steps to reproduce the behavior:

Create a GameObject with ObjectManipulator, BoundingBox and Rigidbody attached
Create a child gameobject with the AppBar class
Try to press any button on the AppBar

Expected behavior
There is no response from the AppBar. It seems like if the raycast from the hands collides with the bounding box collider first, and not with the AppBar button
Your setup

Unity 2019.4.4f1
MRTK 2.4

Target platform

HoloLens 2

Solved it, there was a warning in the bounding box component that was related to this, and the button solved it


Fix instructions to work with Swift 3.0.2

Attempting to follow the instructions with Swift 3.0.2 results in the error: "module compiled with swift 3.0 cannot be imported in swift 3.0.2".
A more robust (though slower) instruction example is: carthage update --platform iOS --no-use-binaries

Thanks @jsstylos for pointing that out! Updated the README with your suggestion.


Doesn't work with Windows Form

It seems that EBind can only bind from my data to Winform Control but cannot bind from Control back to my data.
public partial class MainForm : Form
{
    // auto implement using Fody
    public class Test : INotifyPropertyChanged
    {
        public event PropertyChangedEventHandler PropertyChanged;

        public string Hello { get; set; }
    }

    EBinding binding;
    Test abc = new Test();
    public MainForm()
    {
        InitializeComponent();
        binding = new EBinding {
            BindFlag.TwoWay,
            () => txtHello.Text == abc.Hello,
        };
        abc.Hello = "ABC";
        // txtHello.Text is changed to "ABC"
        txtHello.Text = "DEF";
        // abc.Hello is still "ABC"
    }
}

WinForms controls are platform-specific binding targets which have not been configured out of the box yet.
You need to initialize your app with the configuration code for the control properties that are actually used as a binding source in your project.
In case of a TextBox.Text property the configuration would be:
EBinding.DefaultConfiguration.ConfigureTrigger<TextBoxBase, string>(
    v => v.Text,
    (v, h) => v.TextChanged += h,
    (v, h) => v.TextChanged -= h);

See the configuration docs and for more info.
You might also find a Xamarin.Android config useful as a sample.


merge from template-sync/template-arlac77-github

package.json

chore(deps): add ^24.1.0 remove ^24.0.0 (devDependencies.semantic-release)

.github/workflows/codeql_analysis.yml

chore(actions): merge from template .github/workflows/codeql_analysis.yml

.github/workflows/update_package_lock.yml

chore(actions): merge from template .github/workflows/update_package_lock.yml

.github/workflows/update_readme_api.yml

chore(actions): merge from template .github/workflows/update_readme_api.yml

Pull Request Test Coverage Report for Build<PHONE_NUMBER>2
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 99.219%

Totals

Change from base Build<PHONE_NUMBER>1:
0.0%

Covered Lines:
109

Relevant Lines:
109

💛  - Coveralls


Implement more world noise types & parameters

The current noise for the world outside our sculpted areas is a generic perlin type noise generated in the shader with only scale and height parameters. Offer more noise types and parameters.
Auburn offers his whole library in GLSL which could be #included
https://github.com/Auburn/FastNoiseLite/tree/master/GLSL

Improved world noise added in 289a141823dfed382a8831ee5ed1c8a2765b7eee


glTF sandbox: alpha=0 with MASK mode not transparent

Repro
In https://sandbox.babylonjs.com/, the following glTF material is opaque
Cube-Alpha0.zip
    {
      "alphaMode": "MASK",
      "doubleSided": true,
      "name": "Material.001",
      "pbrMetallicRoughness": {
        "baseColorFactor": [
          0.800000011920929,
          0.800000011920929,
          0.800000011920929,
          0
        ],
        "metallicFactor": 0,
        "roughnessFactor": 0.5
      }
    }

Because this uses MASK mode and the alpha value (0) is less than the alphaCutoff (default of 0.5), this should be fully transparent.
If alpha=0 is achieved with a fully transparent texture instead
Cube-TextureAlpha0.zip
    {
      "alphaMode": "MASK",
      "doubleSided": true,
      "name": "Material.001",
      "pbrMetallicRoughness": {
        "baseColorTexture": {
          "index": 0  // <- this has alpha=0 everywhere
        },
        "metallicFactor": 0,
        "roughnessFactor": 0.5
      }
    }

then it renders transparent as expected.
Screenshots

Desktop (please complete the following information):

OS: Arch Linux
Browser: Firefox
Version: 128

Additional context
https://github.com/KhronosGroup/glTF-Blender-IO/issues/2328

cc @bghgary

We seem to be ending up in alpha blend mode for this case. I will investigate, but we probably need to overhaul how we detect alpha a bit.

I'm going to try to fix this along with #15504 since it's somewhat related, but this may take a while to confirm everything is working.


Is there any way how I can disable DisallowNull generation

Hi.
For specific reason I have an API that has method with parameter and returned value with the same type.
class HelloObject{
[Required]
  string SomeProp {get;set;}
}
----
HelloObject Index(HelloObject) {
}

In this DTO one property is marked with DataAnnotation [Required] for validation of incoming parameter. But output model can have null value there.
But. If I generate definition with NSwag i will get
        [Newtonsoft.Json.JsonProperty("SomeProp", Required = Newtonsoft.Json.Required.DisallowNull, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
        public string SomeProp { get; set; }

And will get an exception while invoking this method in generated Client.
Required property 'SomeProp' expects a non-null value. Path '[0].HelloObject', line 1, position 354.
I've tried:
                    RequiredPropertiesMustBeDefined = false,
                    GenerateDataAnnotations = false,

same as both states of NullHandling
So is there any way how to disable DisallowNull generation?

There is only one schema generated per class. The problem is, that your request and response DTO are different, so you cannot use the same class... The mentioned settings change the behavior for all classes, not just the request or response DTO clases

For me it's ok to have for both of them Required = Newtonsoft.Json.Required.Default.

Here is the code:
https://github.com/RSuter/NJsonSchema/blob/master/src/NJsonSchema.CodeGeneration.CSharp/Models/PropertyModel.cs#L57
do you see a way to support your needs?
i think using required is wrong here because the response can contain null values even if the required attribute is set. this is a violation of the api contract defined in c#. maybe you should just remove the attribute and do manual validation in the operation method?

Seems to be there is no option how can i do it.
I fully agree with you that it should't be like that. But unfortunately I can't change it now, i am not an owner of this code :( But I generated clients for this API. And manually replacing all the Newtonsoft.Json.Required.DisallowNull with Newtonsoft.Json.Required.Default every time when new methods are added.
So would be good to have some way to ignore Required at all. I would do a PR if i saw where we can put this option.

The best option is to transform the generated swagger spec before generating client code (ie with an own cmd line tool which uses the nswag.core lib).

ITNOA
In general I think flexibility in library and tool is important, so I suggest to add this option into NSwag