DB Migrations and Backward Compatibility Issues

This thread keeps track of everything related to DB migrations and backward compatibility issues.

PR #335

System outputs are now stored in Cloud Storage instead of MongoDB
The PR is backward compatible so no DB migration is required.
code & code were added for backward compatibility reasons. They can be removed when we regenerate all system analyses next time.

PR #365

DB migration was performed because I couldn't make the code backward compatible when I upgraded the SDK to v0.11.2.

PR #434 #435 #436

Migration was performed for all three PRs in one go. Please refer to the PRs to see what was changed.

PR #366

DB migration was performed to create a user metadata collection and to replace all system's creator fields from emails to user ids.

PR #439

Migration was performed for dev. Prod DB migration is scheduled for Oct 29, 2022.

PR #490

Migration for both dev and prod performed on Nov 4, 2022.

PR #493 (#494 also related)

Migration for both dev and prod performed on Nov 8, 2022.
sdk_version_used is set to "old" for all systems so all previous systems will get re-analyzed when a user clicks "Analysis".
This migration also partially addresses an issue with custom analyses. custom_features and custom_analyses come from two sources: the dataset (if it is a datalab dataset) or the output file (provided by the user). Because of how the SDK is implemented, the latter always overrides the former if it is present. This migration brings back all custom_analyses of datasets even if a user-defined custom_analyses is present. The code still doesn't work though so new systems still suffer from this issue. When the code is fixed, I can run the migration again to fix all the exiting systems.


Timeout while logging to the Flynn's Docker registry (local, Vagrant)

I have set up a local Flynn cluster using the demo scripts, which seems to work (I can access the dashboard...). I am trying create an application using a docker image, but I can't go past the $ flynn docker login command, the error being:
error running `docker login`: exit status 1 - output: {"Flag --email has been deprecated, will be removed in 1.14.\nError response from daemon: Get https://docker.demo.localflynn.com/v1/users/: dial tcp <IP_ADDRESS>:443: i/o timeout\n" '\x00' "\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00" '\x00'}

It seems that docker can't access the registry, so I have sshed into the docker-machine and checked the logs (via docker-machine ssh dev cat /var/log/docker.log):
time="2017-02-13T17:09:52.926425114Z" level=debug msg="Calling GET /_ping"
time="2017-02-13T17:09:52.927033451Z" level=debug msg="Calling GET /v1.26/info"
time="2017-02-13T17:09:52.942090324Z" level=debug msg="Calling POST /v1.26/auth"
time="2017-02-13T17:09:52.942156562Z" level=debug msg="form data: {\"password\":\"*****\",\"serveraddress\":\"docker.demo.localflynn.com\",\"username\":\"user\"}"
time="2017-02-13T17:09:52.972692090Z" level=debug msg="hostDir: /etc/docker/certs.d/docker.demo.localflynn.com"
time="2017-02-13T17:09:52.973631545Z" level=debug msg="hostDir: /etc/docker/certs.d/docker.demo.localflynn.com"
time="2017-02-13T17:09:52.973669262Z" level=debug msg="attempting v2 login to registry endpoint https://docker.demo.localflynn.com/v2/"
time="2017-02-13T17:10:07.975121478Z" level=info msg="Error logging in to v2 endpoint, trying next endpoint: Get https://docker.demo.localflynn.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
time="2017-02-13T17:10:07.975357542Z" level=debug msg="attempting v1 login to registry endpoint https://docker.demo.localflynn.com/v1/"
time="2017-02-13T17:10:37.976163330Z" level=info msg="Error logging in to v1 endpoint, trying next endpoint: Get https://docker.demo.localflynn.com/v1/users/: dial tcp <IP_ADDRESS>:443: i/o timeout"
time="2017-02-13T17:10:37.976265881Z" level=error msg="Handler for POST /v1.26/auth returned error: Get https://docker.demo.localflynn.com/v1/users/: dial tcp <IP_ADDRESS>:443: i/o timeout"
time="2017-02-13T17:17:28.867844803Z" level=debug msg="Calling GET /_ping"
time="2017-02-13T17:17:28.868919144Z" level=debug msg="Calling GET /v1.26/version"

Strangely enough, it can resolve the docker.demo.localflynn.com address, but it can't access the corresponding ip (I have also tried pinging the address from the docker-machine, without success).
For information, docker version gives:
Client:
 Version:      1.13.1
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   092cba3
 Built:
 OS/Arch:      darwin/amd64

Server:
 Version:      1.13.1
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   092cba3
 Built:        Wed Feb  8 08:47:51 2017
 OS/Arch:      linux/amd64
 Experimental: false

That IP address is for the demo VM, it sounds like there is a network/routing issue between the Docker host and the Flynn VM.

Here are the iptables for the Docker host: Does this look fine to you?
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
DOCKER-ISOLATION  all  --  anywhere             anywhere
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain DOCKER (1 references)
target     prot opt source               destination

Chain DOCKER-ISOLATION (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

(I can ping the Flynn VM just fine from my machine, but not from the Docker host)

This is not an iptables issue, the problem lies somewhere in the host or Docker VM network configuration (probably routing).

I made a new Docker host with a different driver, and it now works.


Using Http on all connections causes some to fail

Entering a URL that only accepts https connections will cause the program to throw an exception.
Stacktrace:
Exception in thread "main" java.io.IOException: Server returned HTTP response code: 501 for URL: http://repo1.maven.org
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1927)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1523)
	at pinger.Pinger.main(Pinger.java:17)

Affecting code: https://github.com/BoogieMonster1O1/pinger/blob/master/src/main/java/pinger/Pinger.java#L14

A suggested approach would be to accept some user input on weather https should be used in the connection, and depending on what was inputted, the resulting url should be modified accordingly.

hello @BoogieMonster1O1 when i implement this , should i remove the instanceof check?


Created Interest Class and Added Progress bar into Register and Login

Working well. I hope this will decrease the number of crashes


Add hermetic grunt tool

The new tool allows every developer to run grunt in the same environment.
This is a "beta" version so people can start playing with it. There are still a few things to fix in follow up changes:

build cache (.tscache) is not reused across runs.
Tests that depend on browsers fail. We need to tweak the image for that.

This change is

Very cool. Thanks for packaging this up!
I think we can interate. For example, all our uproxy-docker images started out "self build" - but now they're published on Docker Hub so that you don't need to build the images every time you want to test something. Can you think about how this image would work if it were pull-able from Docker Hub?

We could publish images that others could use. In that case, the hermetic_grunt.sh script could omit the build command and have something like:
docker run --rm -v $(pwd):/root/repository_root -it uproxy/hermetic-grunt "$@"

With uproxy/hermetic-grunt being the name of the published image.
That could be used by Travis too, for faster test times. And it would solve the problem I'm seeing of intermediate images accumulating in my disk.
The tricky part is that changes to the code may require a change to the image. How does one know when a change affects the image? If images are tied to the code version, which version of the image do you use?
One possibility is to make the hermetic_grunt.sh specify the version of the image. For example: uproxy/hermetic-grunt:9348309284. Whenever we make a change that affects the image, we build and push a new one, update the script, and merge the script change into master. We could place the image version in a file that other scripts (such as the test) can use.
What do you think? I haven't thought it through. We can brainstorm some ideas.

Review status: 0 of 6 files reviewed at latest revision, 11 unresolved discussions.

tools/hermetic_grunt/entrypoint.sh, line 16 [r1] (raw file):
Previously, trevj wrote…
> Several lines here have a traililng `&&` but either have no subsequent command or end with a newline, with no `\` - are you sure they do what you want? I would recommend putting `set -e` near the top of the file so that `bash` will terminate execution on the first error.

You don't need `\` for line continuation when you end it with `&&`.
I use && to tell that the next command should not be run if the previous one failed. You can see it as a ; in Java or C++.
The problem with -e is that you lose control to tell what code should run in case of previous failures.
In this case, I want to always run chown, to make sure we don't end with an out/ directory owned by root.
I now put things in functions, to make the flow a little clearer.

tools/hermetic_grunt/entrypoint.sh, line 18 [r1] (raw file):
Previously, trevj wrote…
> Neat trick. Is there some reason this wouldn't work with `build/`?

It would work for `build/` too. The problem is that `build/` is a mix of inputs and outputs of the build process. Some of the directories come from the image, and some is output by grunt. This `out/` directory contains only the output from grunt, which I overlay over the `build/` from the image.

tools/hermetic_grunt/hermetic_grunt.sh, line 1 [r1] (raw file):
Previously, trevj wrote…
> `#!/bin/bash`

Done.

Comments from Reviewable

Hey! Sorry for the delay on this.
A few comments inline...how about uproxy/build for the image name?
Totally agree Travis could use this, BTW.

Review status: 0 of 6 files reviewed at latest revision, 8 unresolved discussions.

tools/hermetic_grunt/Dockerfile, line 8 [r1] (raw file):
Previously, fortuna (Vinicius Fortuna) wrote…
> FYI, I ended up changing this to `selenium/node-chrome`, which comes with Chrome and makes some tests pass with an additional tweak to `entrypoint.sh`. I copied the trick from the existing `uproxy/Dockerfile`. There are still a few jasmine tests that are failing, but for some other reason.

Oh, cool. I think in time we can get our testing and build images closer to each other.

tools/hermetic_grunt/entrypoint.sh, line 16 [r1] (raw file):
Previously, fortuna (Vinicius Fortuna) wrote…
> You don't need `\` for line continuation when you end it with `&&`.
>
> I use `&&` to tell that the next command should not be run if the previous one failed. You can see it as a `;` in Java or C++.
> The problem with `-e` is that you lose control to tell what code should run in case of previous failures.
>
> In this case, I want to always run `chown`, to make sure we don't end with an `out/` directory owned by root.
> I now put things in functions, to make the flow a little clearer.

I never knew that! Cool.

tools/hermetic_grunt/entrypoint.sh, line 18 [r1] (raw file):
Previously, fortuna (Vinicius Fortuna) wrote…
> It would work for `build/` too. The problem is that `build/` is a mix of inputs and outputs of the build process. Some of the directories come from the image, and some is output by grunt. This `out/` directory contains only the output from grunt, which I overlay over the `build/` from the image.

Interesting. I agree `build/` needs some serious work - FWIW I have some changes planned that should remove all of the inputs from `build/`.

Comments from Reviewable

FYI, I've renamed the image to uproxy/build.
I'm not using the Docker Hub yet, but I plan to use in in a future PR.

Review status: 0 of 6 files reviewed at latest revision, 10 unresolved discussions.

tools/hermetic_grunt/entrypoint.sh, line 18 [r1] (raw file):
Previously, trevj wrote…
> Interesting. I agree `build/` needs some serious work - FWIW I have some changes planned that should remove all of the inputs from `build/`.

That's great! Are you moving that logic to the Gruntfile?

Comments from Reviewable

FYI, I created a new PR that supersedes this one: #2487

Closing this one since I pulled #2487


I don't see how this was any different to the last time you promised something.  You know as well as I do 6 months is not enough to redeem that, and you have proven to me and everyone else that your word has little to no value when it comes to sincerity.
I think you know where this is going.
Your application is declined. Further posts outside this ticket will result in action taken against your GitHub account.
Enjoy your day! :)

I don't see how this was any different to the last time you promised something.  You know as well as I do 6 months is not enough to redeem that, and you have proven to me and everyone else that your word has little to no value when it comes to sincerity.
I think you know where this is going.
Your application is declined. Further posts outside this ticket will result in action taken against your GitHub account.
 I have this fantasy where we start talking at the Vanity Fair Oscars party bar. We exchange a few pleasantries. She asks what I do. I say I loved her in New Girl. She laughs. I get my drink.
"Well, see ya," I say and walk away. I've got her attention now. How many guys voluntarily leave a conversation with Natalie Portman? She touches her neck as she watches me leave.
Later, as the night's dragged on and the coterie of gorgeous narcissists grows increasingly loose, she finds me on the balcony, my bowtie undone, smoking a cigarette.
"Got a spare?" she asks.
"What's in it for me?"  She smiles.
"Conversation with me, duh."
I laugh.
"What's so funny?" she protests.
"Nothing, nothing... It's just... don't you grow tired of the egos?"
"You get used to it," she says, lighting her cigarette and handing me back the lighter.
"What would you do if you weren't an actress?" I ask.
"Teaching, I think."
"And if I was your student, what would I be learning?"
"Discipline," she says quickly, looking up into my eyes, before changing the subject. "Where are you from?"
"Bermuda," I say.
"Oh wow. That's lovely."
"It's ok," I admit. "Not everything is to my liking."
"What could possibly be not to your liking in Bermuda?" she inquires.
"I don't like sand," I tell her. "It's coarse and rough and irritating and it gets everywhere."

Natalie Portman is the reason I work out. I have this fantasy where we start talking at the Vanity Fair Oscars party bar. We exchange a few pleasantries. She asks what I do. I say I loved her in New Girl. She laughs. I get my drink.
"Well, see ya," I say and walk away. I've got her attention now. How many guys voluntarily leave a conversation with Natalie Portman? She touches her neck as she watches me leave.
Later, as the night's dragged on and the coterie of gorgeous narcissists grows increasingly loose, she finds me on the balcony, my bowtie undone, smoking a cigarette.
"Got a spare?" she asks.
"What's in it for me?"  She smiles.
"Conversation with me, duh."
I laugh.
"What's so funny?" she protests.
"Nothing, nothing... It's just... don't you grow tired of the egos?"
"You get used to it," she says, lighting her cigarette and handing me back the lighter.
"What would you do if you weren't an actress?" I ask.
"Teaching, I think."
"And if I was your student, what would I be learning?"
"Discipline," she says quickly, looking up into my eyes, before changing the subject. "Where are you from?"
"Bermuda," I say.
"Oh wow. That's lovely."
"It's ok," I admit. "Not everything is to my liking."
"What could possibly be not to your liking in Bermuda?" she inquires.
"I don't like sand," I tell her. "It's coarse and rough and irritating and it gets everywhere."

whats this supposed to mean

whats this supposed to mean

It means that we no longer care so I can use copypastas in an attempt to be funny.


Allow for additionalTextEdits in completion items.

Is your feature request related to a problem? Please describe.
Current the built-in lsp support doesn't take into account that completions may have additionalTextEdits in them. I created a feature request for this already in nvim, and you can see the suggestion for a work around here. When playing around with it I realized that you already have something pretty similar to this with your confirmCompltion() function.
https://github.com/haorenW1025/completion-nvim/blob/7aedc744e2c7cd615a60b7b59d3e3c6a5491154e/lua/completion.lua#L51-L67
It'd be cool to also have this check for additionalTextEdits, and if so, apply them.
Describe the solution you'd like
Piggyback off confirmCompletion() and account for additionalTextEdits.
Describe alternatives you've considered
Locally listen for CompleteDone and have a function locally that does this.

Yeah that's definitely possible. I'll try to implement this.

I've shamelessly used the code snippet from the discussion you mentioned and it should be working now, please update and check:)

Cool, so I just gave this a try on the following completion:
    {
      "label": "Instant - java.time",
      "kind": 9,
      "detail": " java.time",
      "sortText": "00004",
      "filterText": "Instant",
      "insertTextFormat": 1,
      "textEdit": {
        "range": {
          "start": {
            "line": 10,
            "character": 10
          },
          "end": {
            "line": 10,
            "character": 16
          }
        },
        "newText": "Instant"
      },
      "additionalTextEdits": [
        {
          "range": {
            "start": {
              "line": 1,
              "character": 0
            },
            "end": {
              "line": 1,
              "character": 0
            }
          },
          "newText": "\nimport java.time.Instant\n"
        }
      ],
      "data": {
        "symbol": "java/time/Instant#",
        "target": "file:/Users/ckipp/Documents/scala-workspace/metals-minimal-sbt/?id\u003droot"
      }
    }

And I get the following error:
|| Error detected while processing CompleteDone Autocommands for "<buffer=3>":
|| E5108: Error executing lua ...rs/ckipp/.vim/plugged/completion-nvim/lua/completion.lua:59: attempt to perform arithmetic on upvalue 'lnum' (a table value)

It should be fixed now(I've checked with the given completion), please update and check.

Yes! This indeed does work. Thanks so much for implementing this! I appreciate all the work you are putting into this plugin 👍

No problem:) I'll also make an option for this and having it on by default in case people doesn't want this feature.


use different IDs for both session-sides

Both sides of a session should be allowed to have different IDs and as replacement, both should store the id of the other side. This would result in the advantages:

no pending list anymore
solve the race-condition of #11
no id-change-messages necessary

changed the session-transfer, but a little different. both sides still have the same id. Its a 32 bit unint, which is the combination of the 16 bit id from the client and the 16 bit id from the server. 16 bit are enough for over 32k sessions.


Do not show (resource_)filegroups not belonging to focused targets

Background
Addresses https://github.com/MobileNativeFoundation/rules_xcodeproj/issues/1748
Change
Per https://github.com/MobileNativeFoundation/rules_xcodeproj/issues/1748:

associate owner to resources in internal/resources.bzl, instead of just the file_paths.
process those in xcodeproj_rule.bzl, filter those files to only ones in focused_labels.

According to the errors, it looks like I might need to change more.
focused_targets_extra_files.append(xcode_target.inputs.resources)

type of (xcode_target.inputs.resources) = 'depset(tuple(owner,resource))'
and from
        for file, owner_label in owned_extra_files.items():
            if label_str == owner_label:
                focused_targets_extra_files.append(
                    (
                        label,
                        depset([file.path for file in file.files.to_list()]),
                    ),
                )

we are adding entries of type tuple(Label, depset(String))
I think it would be more aligned if we make xcode_target.inputs.resources to the same type tuple(Label, depset(String))

@narlei can you check if my change works on your end? I have checked on our end that it's working fine.

@narlei can you check if my change works on your end? I have checked on our end that it's working fine.

Hi, I tested with your branch, and it worked as well.
Using hyperfine, I did a performance test and I didn't notice significant changes.


feat: Implement tower-service::Service instead of GrpcService

InterceptedService requires that client implement Service trait from tower-service rather than GrpcService from tonic for some reason.
GrpcService has blanked implementation for anything that implements Service with “correct” bounds.
Current implementation in grpc-web-client prevents using it with interceptor due to that. This PR address that.
Verified that change has no regression on my private project.

I've created a crate based on this with updated dependencies: https://crates.io/crates/tonic-web-wasm-client
Also, in tonic-web-wasm-client, Client implements Service instead of GrpcService.


Adds generic damage visualizer to the game

About the PR
This PR adds a generic damage visualizer to the game. What this means is, rather than create a new visualizer for damage, or anything else in order to display how much damage an entity has taken visually (e.g., brute, burn, whatever), you can instead add this to an entity's visualizers within its appearance component, and feed it DamageSpecifier data to get automatic, even damage display.
How
The visualizer takes these fields:

thresholds: required by the visualizer. This is what triggers a damage visual state change between each damage update. All damage groups will follow this set of thresholds - negative thresholds are invalid, and it automatically hides all damage when the threshold hits zero.
overlay: This toggles if the visualizer should either overlay the sprite, or replace the sprite's states otherwise.
targetLayers: This is a set of sprite layers to target - this works if these were statically defined in the entity's prototype. This is completely optional if overlay is set to true, as it will otherwise just add in a blank layer at the very top. If overlay is set to false, however, it is not optional, as the visualizer must target something to change when damage occurs. These can also be set to be disabled visually by passing in that sprite layer's key into the AppearanceComponent's SetData, and setting its value to false.
damageSprites: This is a dictionary where the values are paths to RSIs containing damage overlays, and the keys are DamageGroup prototype IDs. If the visualizer's overlay is set to true, this cannot be null, and if the visualizer's overlay is set to false, this must be null.
damageGroup: This is a string consisting of a single damage group to target. This only takes effect if the visualizer's overlay is set to false (the reasoning behind this is that the amount of sprites to create for combined damage groups seems excessive - but if anybody has any way to make that better, I would appreciate the help!) If this is defined with damageSprites, then damageSprites will always take precedence.

Sprites in every damage overlay RSI, or in every base sprite's RSI have these forms:
{original layer state}_{damage group}_{threshold}
For example: r_foot_Brute_10 if you want a layer with a state r_foot to have an overlay if it has above 10 Brute damage.
Alternatively, if you're not targetting any layer on the sprite, the sprite's name has this form: DamageOverlay_{damage group}_{threshold}
What for?
This is mostly filler for any object that needs a damage visualizer, without having to write more code than needed. If an entity's damage visualization case requires that it needs a complex visualizer attached to it, then this isn't very useful for that use case.
Screenshots

I don't have many useful screenshots, since I'm not that good at spriting, and this is more of a dynamic sprite layer change if anything. I can take video if needed, however, to showcase how this works.
Changelog

:cl:

add:  NanoTrasen has cut the budget on maintaining how pretty things look on our stations. You may notice things around you slowly become more decrepit over time. Examples of this include walls, vending machines, and skin.

based, would like if there were some example screenshots/video of it working

one, I don't know how actually useful this is

Very, we need it for windows and body parts for example.

one, I don't know how actually useful this is

Very, we need it for windows and body parts for example.

actually, body parts were one of my primary reasons for doing this - you can't specifically target layers (at the moment!) and give them damage, though, instead the damage is applied evenly throughout the entire body according to the internal DamageSpecifier that the humanoid has

actually, body parts were one of my primary reasons for doing this - you can't specifically target layers (at the moment!) and give them damage, though, instead the damage is applied evenly throughout the entire body according to the internal DamageSpecifier that the humanoid has

I'm so sorry that you had to interact with body code, as one of the few people that have been in charge of it.
The sprites that we intended to use for damaged body parts are in #2022, if they are of any use to you.

I noticed a big flaw with this visualizer. It stores some per-entity data in itself. Visualizers are all meant to be flyweighted per prototype, meaning that one visualizer instance controls how many entities with the same prototype look. If you want to store some special per-entity appearance data, you should probably do so with AppearanceComponent data instead of storing it in the visualizer

sure - so anything that i assumed would've been stored as part of an instance of a visualizer (e.g., what sprite layers have been disabled on an entity, what the last damage threshold was for an entity) should instead be stored in the visualizer's internal data?

i'll try to update this with comments to make it more clearer!


Release asset not found when triggered from release event

Behaviour
Expected behaviour

When triggered from release event and adding a release asset to the release in a previous step, the action should be able to download the asset and send it to VirusTotal for analysis.

Actual behaviour

The action sees the added file but cannot download it, and fails with a "(85874065): Not Found" error. I can confirm that the file has been added to the release since I can download the file manually from the release page after the workflow has run. Also, this same workflow works when triggered manually (with slight adjustments to account for having to create the release in the workflow), it just seems to fail with this error when triggered by publishing a new release. I've also tried adding a "sleep" step before crazy-max/ghaction-virustotal to account for any delay in the file being available to download but this made no difference.

Configuration
name: Release

on:
  release:
    types: [published]

jobs:
  release:
    name: Update GitHub Release
    runs-on: windows-latest
    steps:
      - uses: actions/checkout@v3

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 'lts/*'
          registry-url: https://registry.npmjs.org

      - name: Download artifacts
        uses: actions/download-artifact@v3
        with:
          path: artifacts

      - name: Update release
        uses<EMAIL_ADDRESS>        with:
          allowUpdates: ${{ true }}
          artifacts: 'artifacts/release.zip'
          tag: ${{ env.GITHUB_REF_NAME }}
          token: ${{ secrets.GITHUB_TOKEN }}

      - name: Upload adapter zip file to VirusTotal
        if: '!github.event.release.prerelease'
        id: virustotal-scan
        uses<EMAIL_ADDRESS>        with:
          files: |
            .zip$
          vt_api_key: ${{ secrets.VIRUSTOTAL_API_KEY }}

Logs
2022-11-24T23:22:16.3312894Z ##[group]Run<EMAIL_ADDRESS>2022-11-24T23:22:16.3313303Z with:
2022-11-24T23:22:16.3313510Z   allowUpdates: true
2022-11-24T23:22:16.3313905Z   artifacts: artifacts/release.zip
2022-11-24T23:22:16.3314232Z   name: v1.4.0-beta.1
2022-11-24T23:22:16.3314459Z   omitPrereleaseDuringUpdate: true
2022-11-24T23:22:16.3314909Z   token: ***
2022-11-24T23:22:16.3315119Z   generateReleaseNotes: false
2022-11-24T23:22:16.3315329Z   omitBody: false
2022-11-24T23:22:16.3315539Z   omitBodyDuringUpdate: false
2022-11-24T23:22:16.3315775Z   omitDraftDuringUpdate: false
2022-11-24T23:22:16.3316075Z   omitName: false
2022-11-24T23:22:16.3316428Z   omitNameDuringUpdate: false
2022-11-24T23:22:16.3316649Z   removeArtifacts: false
2022-11-24T23:22:16.3316864Z   replacesArtifacts: true
2022-11-24T23:22:16.3317086Z   skipIfReleaseExists: false
2022-11-24T23:22:16.3317879Z   updateOnlyUnreleased: false
2022-11-24T23:22:16.3318110Z env:
2022-11-24T23:22:16.3318338Z   NPM_CONFIG_USERCONFIG: C:\actions-runner\_work\_temp\.npmrc
2022-11-24T23:22:16.3318612Z   NODE_AUTH_TOKEN: XXXXX-XXXXX-XXXXX-XXXXX
2022-11-24T23:22:16.3318820Z ##[endgroup]
2022-11-24T23:22:27.9887938Z ##[group]Run<EMAIL_ADDRESS>2022-11-24T23:22:27.9888713Z with:
2022-11-24T23:22:27.9889212Z   files: .zip$
2022-11-24T23:22:27.9890198Z   vt_api_key: ***
2022-11-24T23:22:27.9890741Z   vt_monitor: false
2022-11-24T23:22:27.9891293Z   monitor_path: /
2022-11-24T23:22:27.9891887Z   update_release_body: false
2022-11-24T23:22:27.9892767Z   github_token: ***
2022-11-24T23:22:27.9893296Z   request_rate: 0
2022-11-24T23:22:27.9893989Z env:
2022-11-24T23:22:27.9894685Z   NPM_CONFIG_USERCONFIG: C:\actions-runner\_work\_temp\.npmrc
2022-11-24T23:22:27.9895532Z   NODE_AUTH_TOKEN: XXXXX-XXXXX-XXXXX-XXXXX
2022-11-24T23:22:27.9896175Z ##[endgroup]
2022-11-24T23:22:28.3972094Z Release event detected for refs/tags/v1.4.0-beta.1 in this workflow. Preparing to scan assets...
2022-11-24T23:22:28.7701951Z 1 asset(s) will be sent to VirusTotal for analysis.
2022-11-24T23:22:28.7714764Z ##[group]1 asset(s) will be sent to VirusTotal for analysis.
2022-11-24T23:22:28.8623900Z ##[endgroup]
2022-11-24T23:22:28.8716486Z ##[error]Cannot download release asset release.zip (85874782): Not Found

Add a sleep of about 10s between actual release event and trigger of this virus total action.


MicroK8s in Raspberry Pi 5 with Raspberry Pi OS 64 Bit - node not ready

Summary

What Should Happen Instead?

Reproduction Steps

Install raspberry pi os on pi5
Enable cgroup according to the documentation on new line in /boot/firmware/cmdline.txt
Reboot
Install snapd (sudo apt install snapd)
Reboot
Install snapd (sudo snap install snapd)
Sudo snap install microk8s —classic
Sudo microk8s start

Introspection Report

Not sure . If I am doing something wrong here
Can you suggest a fix?

Are you interested in contributing with a fix?

Could you please attach a microk8s inspect tarball?


[REEF-573] Update Hadoop dependency version to 2.6.0

JIRA: REEF-573

I'll test this

Tests on HDInsight cluster pass. This is sufficient validation for this change, right?

Yes, that should do the trick


Odoc parser does not render comments properly

In the current state of affairs only the first line of the comment is treated as such, the rest is rendered as is

All grammar is ignored, even in the first line.
Compare this to working odoc parser in ocaml-platform

Demo:
https://github.com/Lomand/rescript-broken-autocomplete-jsx/blob/master/src/Demo.res

Odoc comments are not supported. Doc comments are getting gradually converted to markdown:
https://github.com/rescript-lang/rescript-compiler/blob/2f0201f471df94dc4650966132c011b97ff09a1f/jscomp/others/belt_List.mli#L241

Ok, looks like anyone who wants to generate docs from the source code is out of luck. Such a shame

Yeah the docgen stuff I wrote extracts all the markdown comments into a JSON file that can actually be processed outside of OCaml tooling... a huge issue we originally had with odoc back then.
One thing that's still missing though is semantic /**/ parsing, so we currently rely on @ocaml.doc as our comment marker still.

I think parsing /** */ should be easy if we agree on its syntax.
i.e
/** {content} */ -> multiline comment and annotate with ocaml.doc (or whatever).
/* {content} */ -> multiline comment

Closing this as I believe this either isn't an issue anymore, or is going to be fixed whenever core modules are converted to res + markdown.


Can this plugin be used on linux machine?

I have tried the plugin on windows and it works just fine. However we are running into issues with few folders checked in in RTC through the browser  where there is trailing white space in folder names. Would like to try the plugin on linux platform to see if the changesets associated with these folder names get accepted successfully.
I have installed eclipse and configured SCM Path in my linux server. The plugin is copied under /jazz/scmtools/eclipse/plugins . However the migrate-to-git command is not enabled. Any suggestions would be helpful.

Hello @rambaliga, as far as i know is the rtc2gitcli not tested with the linux scmtools. Did you installed the Linux Version of SCM Tools?

@fabieu back then, when we migrated all our source we let the complete migration run within a Linux docker image. Therefore yes, at least at one point the Linux SCM Tools where tested. There are still some docker images available here: https://hub.docker.com/repository/docker/rtcto/rtc2git

Its actually more advisable to do the migration on linux, because it can handle case sensitive commits (windows cant). However rtc2gitcli wasnt tested there for a while. But the docker image should still work I think


Add example usage for element (as needed) in the model

This will be to add example usage for element as needed, in the model.

I have added simple textual examples for the following:

WhereClauseCondition class
groups slot
context slot
sectionType slot

I have added example YAML content for the following:

WhereClause class
WhereClauseCompoundExpression class


variant2 mapping metrics

Hi,
Question about  'offtarget' in metrics.tsv generated with variant2 pipeline. Is it enough that part of the read is off target for it to be counted or does the entire read have to be off target?
Linda

Linda;
Off-target (and other coverage statistics) are calculated relative to bases covered, not with respect to reads. So if a read half overlaps on-target and off-target the off-target section will be counted. Hope this helps.

Yes, thank you!


AssertionError in border_correction

Hi,
Feature detection on one of my datasets raised the following error:
Traceback (most recent call last):
File "C:\Users\macraild\progs\peakonly\gui_utils\threading.py", line 72, in run
result = self.function(*self.args, **self.kwargs)
File "C:\Users\macraild\progs\peakonly\processing_utils\runner.py", line 147, in call
features = self._batch_run(files, progress_callback, operation_callback)
File "C:\Users\macraild\progs\peakonly\processing_utils\runner.py", line 261, in _batch_run
border_correction(component, borders)
File "C:\Users\macraild\progs\peakonly\processing_utils\run_utils.py", line 326, in border_correction
assert np.abs(counter_order[-1] - counter_order[-2]) == 1, "almost impossible case :)"
AssertionError: almost impossible case :)
Simply commenting out the offending assert allowed processing to proceed, and I can't see any obvious errors in  the results, but I dont understand enough of what is going on here to know if this is a safe workaround. Please let me know if there is anything I can do to help track down this bug, but also if this can be safely ignored, or if it points to a bigger problem.

Thanks, changing the minimum ROI length seems to avoid the bug.


Explore GitHub GraphQL API Early Release

Might be helpful for this project
See: https://github.com/HumanDynamics/law.MIT.edu/issues/23
and: https://developer.github.com/early-access/graphql

This is now in the file where we are listing tech to check out, here:
https://github.com/HumanDynamics/RevolvingLoanHack/blob/master/ProjectPlan/RelevantTechnologyAndData.md


Search bar

This PR:
-The SearchAOIToolbar directly accesses and dispatches actions to the store instead of receiving two instances of the store from two different parents, thus causing issues with the store getting properly updated.
-Changed the initial state for the geocodeReducer's 'fetching' and 'fetched' from 'null' to 'false'. This pattern is implemented in the other reducers where any fetching is being made and is proper convention according to redux docs -actions are typically set to be booleans where as the objects are set to be nullified or defined.

Most of the logic was set up for error handling, no retrieval of data, and loading a spinner when fetching data. The issue was the way it was talking to state as mentioned in first comment.
Had to update/override react-bootstrap styles for the typeahead and menu components and for the spinner as the display was turned off and other styles were causing issues.
Updated all tests to support revised state handling.

This branch will be deleted as it was built on a non-master branch. Moved all work to the searchBarRevised branch.


Lockscreen notification isnt displayed

I blocked the original music notification by using gravity box,and I found I cannot control music in lockscreen because medianotification dont have a control notification displayed in lockscreen
I believe It will be better to let users to control music in lockscreen with medianotification's build-in lockscreen control notification.
Thanks

Could you explain what you mean by a "control notification"? On stock android, all notifications should be visible on the lock screen by default. I don't know what you mean by a "build-in lockscreen control notification".

I mean when I block the original app's notification,there will be no notification displayed on the lockscreen(both original app and medianotification) and user cannot pause music on lockscreen.
Maybe it is my ROM's issue

I cannot upload screenshots for some reasons...sorry
I just wonder if it works fine on your test phone

@TheAndroidMaster
Well I found the medianotification's notification will be visible after clicking Phonograph's notification on lockscreen.
In other words medianotification's notification will not be shown automatically on lockscreen,it need me to press phonograph notification's pause button to "active" it.
I think it maybe a bug cause the music app on my phone can show notification on lockscreen automatically but medianotification can't.
Thanks very much,and please forgive my poor grammar.

Maybe I know why.
I found medianotification's notification isnt dismissible even if I turn on the "Always make notification dissmissible" button,and android will not show instant notification again on lockscreen.That's why I can only see medianotification's notification once.
So I think it's might be a bug of application.


Confirm Add vs. Create usage

Is the recommendation on usage of Create vs. Add usage clear? Seems right, but should confirm.

@kefo can you confirm we're reflecting the conversation about adding an entity to a particular scheme vs. creating a new entity?

It's not, but it may be that we should alter how we view this.
So we view Add, Update, and Remove activity types in relation to the scheme.  X is Added to the Scheme; X is Updated in the Scheme; X is Removed from the Stream.  When we use "Add," the concept/term/thing was, in fact, newly created.  It wasn't embargoed, or temporarily made unavailable, and there was no change in permissions.  The Add/Update/Remove pattern is just a different viewpoint of the same thing, essentially, being expressed by the Create/Update/Delete pattern.
To me, the Add/Update/Remove pattern was attractive because of "Remove."  The concept of Deletion in linked data raises the fears that URIs are not persistent, that they will not always be available.  Perhaps that is a little too literal - and impractical - because things will be deleted and will disappear forever (Fedora - an LDP implementation - does this, though it leaves a tombstone in its place) but it's not that easy when dealing with authority/vocabulary data, like we do at ID.LOC.GOV.  If someone uses a Name URI and then that Thing is discovered to be a duplicate, we can't just stop serving the URI (or we can, but that's bad form), especially if we can, maybe, point to the correct/active URI in such a scenario.  So we do not delete the Thing; we still honor the URI.  Instead, we deprecate the resource and we remove it from the scheme.  (This was our pattern for handling deletion/removal/cancellation before ActivityStreams.)  And that's why the Add/Update/Remove pattern was attractive to us (and they were, of course, AS types).  But we can alter what we do for the sake of conformity so long as there is a general understanding that "Delete" does not imply Gone.

I don't know. I think we could simplify the delete section, https://ld4.github.io/entity_metadata_management/api/0.1/recommendations.html#delete-entity. I like the idea of only using delete if something is truly deleted (and we discourage Deleting), encouraging instead Create/Add/Update/Remove.

@kefo I just reread that the Create vs. Add section, and I think we meant to think of Add in terms of scheme or vocabulary, but it doesn't read that way. I think we (I) should try to make clearer the distinction between stream and scheme.

Original create vs add text was from https://github.com/LD4/entity_metadata_management/commit/8904a6b637ff967eb797a11c7496ff9fe99b8c92


Prevent PartialEvaluator_buildFormXObject from failing if the Matrix or BBox contains indirect objects

Add getArray and getAsyncArray methods to Dict

These methods extend get, respectively getAsync, and will fetch all indirect objects (i.e. Refs) when the result is an Array.

Prevent PartialEvaluator_buildFormXObject from failing if the Matiix or BBox contains indirect objects

This patch fixes yet another instance of bad PDF data, specifically a case where the BBox array contains indirect objects (i.e. Refs).

Fixes the missing image in http://www.int.washington.edu/talks/WorkShops/int_08_37W/People/Franz_M/Franz.pdf#page=24. Note: There are missing images on a number of the pages in that file.

Please note: I've not had time to create a reduced test-case yet, hence the WIP-label.

/botio-windows preview

/botio test

/botio test

/botio-linux preview

/botio test

/botio makeref

Good work!


Bug fix empty search field

Summary of Changes
Include a summary of changes then remove this line
Tests
What tests did you add or modify and why? If no tests were added or modified, explain why. Remove this line
Documentation
What documentation did you add or modify and why? Add any relevant links then remove this line
CheckList
Make sure you have checked all steps below to ensure a timely review.

[ ] PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"

In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

[ ] PR includes a summary of changes.
[ ] PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
[ ] In case of new functionality, my PR adds documentation that describes how to use it.

All the public functions and the classes in the PR contain docstrings that explain what it does

[ ] PR passes make test

Does this PR resolve https://github.com/lyft/amundsen/issues/243?

@jornh  I think yes.

Codecov Report

Merging #90 into master will decrease coverage by 0.27%.
The diff coverage is 50.00%.

@@            Coverage Diff             @@
##           master      #90      +/-   ##
==========================================
- Coverage   82.02%   81.74%   -0.28%
==========================================
  Files          14       14
  Lines         584      586       +2
  Branches       64       65       +1
==========================================
  Hits          479      479
- Misses         85       86       +1
- Partials       20       21       +1

Impacted Files
Coverage Δ

search_service/proxy/elasticsearch.py
77.31% <50.00%> (-0.73%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5256aa2...8635fe0. Read the comment docs.

thanks


Add iFedAvg framework

Features

Created a custom layer class InteroperabilityLayer that implements the personnalization operations from David Rochewitz's paper.
Training script can now train a model in a distributed manner, using the interoperability framework. (It takes a given model and wraps it between two InteroperabilityLayers)
Create a class for each type of personalized model, that wraps a TF.js model inside an object and allows us to have methods for creation as well has getting the whole model or just the personalized part of it.
Add communication with the server to push your personal interoperability parameters to the server and query the aggregation parameters from all other clients.
Add Heatmaps for Interoperability Interpretation.

UI Change

Added a pannel where one can select Distributed Training Options.
Add a heatmap in the Training information panel.

This should be woking now, I am waiting for Hugo's Synchronized training PR to be up to do the communication.

I am switching this to Ready for review.
There are still some changes I need to do in order to make it work correctly in the synchronous training, but it is WIP.
I think you guys can already give a look at the code as most of it is here.

The work of iFedAvg needs to merged into the experimental branch, and so a separate PR is necessary.


checkov: 2.0.1076 -> 2.0.1077

Description of changes
https://github.com/bridgecrewio/checkov/releases/tag/2.0.1077

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 170093 run on x86_64-linux 1

1 package built:

checkov


References updates

This PR makes a few updates to the way references are tracked in order to support some additional features:

Automatically number references based on the order in which the inline Cite/Sidenotes are rendered

Previously, references.js numbered the references based on the order in which they were threaded through to ReferencesProvider. This (1) relied on object ordering and (2) required consumers to manually sort the references object.

Add a PrintedFootnotes section to the bottom of Article which displays a list of references using the Endnote layout (isolated to printing using '@media print')

Previously, citation contents were hidden on print because of the mobile styles that are applied.
Addresses references point in https://github.com/carbonplan/research/issues/206.
See example PDF (also generated using scrambled references and two inline Sidenotes).

Support additional, notes added inline via the Sidenote component

Allows non-citation notes to be easily added in text. For example, adding:...or macronutrient fertilizers.<Sidenote>Sample custom note defined inline.</Sidenote>

to this line in the seaweed farming article would render:

closes https://github.com/carbonplan/research/issues/205

This looks really clean to me! And super excited for this functionality.  I'm not sure we'll have this often (or ever), but I seem to recall we've avoided it because we were worried it would screw things up.
From a rendering standpoint, I think what we want is if <Cite id='example'/> appears, say, second, it has the superscript number 2 and we have the full citation to the right, and then if <Cite id='example'/> appears  again later on in the article, the superscript number 2 appears but we don't repeat the full citation again (because we have it above), and the numbering continues as though there was no repeat. Is that easy to support given this implementation?

Ah good question! I added in a hide prop, which will allow a repeated citation to be omitted from the inline references. This will have to be added manually for repeated references, which I think is desirable because we currently don't handle scrolling  to references.
Also while I was touching the logic that handles groups of IDs, I tweaked the mobile rendering so that all references can be opened inline:

Previously, this was rendered using 3-6 which we can still use for desktop + printing (see updated example):

lgtm! and very nice tweak for mobile, that's smart


Update sbt-mima-plugin to 0.6.4

Updates com.typesafe:sbt-mima-plugin from 0.6.1 to 0.6.4.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.typesafe", artifactId = "sbt-mima-plugin" } ]

labels: sbt-plugin-update, semver-patch

Coverage remained the same at 89.873% when pulling 87378455b175f6cf7bfc528c0b740f3cd2e0647a on scala-steward:update/sbt-mima-plugin-0.6.4 into b24643ef92e0f5d35696c328da6d03875f02da6e on sangria-graphql:master.

Updated with https://github.com/sangria-graphql/sangria-ion/commit/cc14586b0d41ad31ddb03c2acc111cba86b918d3


css src write after html src filename do not be replaced

cacheBust: {
  options: {
  },
  assets: {
    files: [{
      expand: true,
      cwd: '<%= config.dist %>',
      src: ['page/**/*.html' ,'css/**/*.css']
    }]
  }
}

When I write config like this, the image file names in style sheets files do not be replaced, but when I write like below it works:
cacheBust: {
  options: {
  },
  assets: {
    files: [{
      expand: true,
      cwd: '<%= config.dist %>',
      src: ['css/**/*.css', 'page/**/*.html']
    }]
  }
}

That's a strange bug. Could you try using the v1.0.0-beta branch and see if that fixes the problem. After a few people have tested the branch I'll get it merged in.

I downloaded the v1.0.0-beta,but it did not work.
Cannot read property 'indexOf' of undefined

Thanks for trying. Was that the complete error given back? Did it give you back a stack trace?

Possibly fixed when fixing #173 - needs re-testing

I've tested this myself and can no longer see the issue. Let me know if it comes back


Question: Can We use it without redis ?

Is it possible to use this package without redis dependecy ? I can see that it also supports http, can we use for example curl to broadcast messages to client-side without using redis ?
Thanks

Yes

@barryvdh : I assume http can be used for that as the docs say but can you give a simple example of how to send such request? Will event class still be needed ?

You can just use events with the sync queue driver. See https://github.com/tlaverdure/laravel-echo-server/blob/master/README.md#http

@barryvdh : Thanks yes i had seen that. My question is how to send that POST request with laravel as the docs say:
Request Endpoint
POST http://app.dev:6001/apps/your-app-id/events?auth_key=skti68i...
Request Body
{
  "channel": "channel-name",
  "name": "event-name",
  "data": {
      "key": "value"
  },
  "socket_id": "h3nAdb134tbvqwrg"
}

So it is still a bit confusing. Docs do not give example how to send http POST request with event class.
Thanks

You can use the pusher api and examples from laravel docs, but with the socketio endpoint/user etc

Doesn't help, closing anyway...

What did you try? Did you set your broadcasting driver to pusher, following https://github.com/tlaverdure/laravel-echo-server#pusher ?

yes I set but I don't know how to send http POST with event class as the docs suggest.

@barryvdh : Thanks one last question. Can we send events without pusher ? (You see I am trying to avoid redis or any third party services)

The HTTP API simulates the Pusher API. If you configure the pusher options to use localhost (or whatever server it runs on) with the socketio port, you are not using Pusher, just the HTTP api. So Laravel will take care of sending the HTTP post messages to SocketIO. If you configure your queue driver as sync, you don't need redis.
You can also create your own driver but creating the http messages with Guzzle or something, but would not recommend that.

@barryvdh thanks now it makes sense, thanks for the help.

@barryvdh thanks for all the help, I managed to use this package now without redis, pusher.com and queues.

I cant get it working. Events are firing, channels are up. I see the fired event and its payload in laravel.log but i can't get it to the frontend. What are the settings for using the pusher-options to work with the http api?
Do i only have to insert my key and app_id generated by laravel-echo-server and the options host, port and scheme in the connections array?
If i do that and trie to start the server i get "Class 'Pusher\Pusher' not found". Do i have to install it
even though i wont use it?

I implement RabbitMQ as Laravel broadcasting driver. Check it: https://github.com/hungntv/laravel-echo-server-with-rabbitmq
I had a mission, deploy a laravel project on Windows Server where can't install Redis officially. That's why I make my github repo. I'm still develop it. Feel free to ask me.


[Barefoot Coders] Partner Information

Organization name
Barefoot Coders
Brief description of your business and what it does (50 words or less)
Barefoot Coders is a boutique software and technical infrastructure development studio, helping organizations scale through cloud-native solutions.
Name of your relevant product(s) or service for the OpenSearch audience
Our comprehensive cloud architecture, engineering, and support practice offers organizations of all sizes a one-stop shop for establishing a foundation for, migrating to, managing, and optimizing within public cloud environments. Organizations that require comprehensive search, logging, analytics and observability will benefit from our domain expertise with OpenSearch in AWS.
What OpenSearch technologies do you specialize in
Search & Log Management
Type of business
Professional Services Organization
Home/main office location or address
New York City: 240 Kent Ave. Ste B19, Brooklyn, NY 11249
Portland, OR: 830 NE Holladay St, Portland OR 97232
How can people contact you (email and phone)
Email<EMAIL_ADDRESS>Phone: +1<PHONE_NUMBER>
Regions you serve
North America
Industry specialization
N/A - Industry & vertical agnostic
Logo

Additional image
We're working on this -- is there a timeline that we need this by?
Links
https://www.barefootcoders.com/ -- we're working to build our blog presence later this year.

@Jberlinsky - Thank you for submitting the info.


new advanced version

Hi ,would you pls let me know how to switch to advanced version after deploy local ?
will you support new LLm llmada3.1 ?
Thank you

To switch to advanced, simply visit /advanced. Advanced page
Llama 3.1 will be supported in the future when the model is out of preview release.


Fix disk block cache time decoding error

Updated go-codec encodes time.Time values in a different way,
which was breaking disk block cache. Use the old encoding method
instead.

PTAL @jzila!


Issue ([Errno 54] Connection reset by peer) with PokemonOptimizer task

Expected Behavior
The evolve process continue without any issue.
Actual Behavior
Evolve process returns an Error ([Errno 54] Connection reset by peer) every few evolve. Please see the logs: http://pastebin.com/1HU82FM3
Same thing happens during transfer:
http://pastebin.com/Xmfbb086
Your FULL config.json (remove your username, password, gmapkey and any other private info)
http://pastebin.com/LjBKMwhC
Output when issue occurred

http://pastebin.com/1HU82FM3
http://pastebin.com/Xmfbb086
Steps to Reproduce
Use the task PokemonOptimizer to transfer or evolve.
Other Information
OS:
Mac OS X 10.11.6
Branch:
dev
Git Commit:
f0df954ee0e79c1dbfd3be76324b73b36988915b
Python Version:
2.7
Any other relevant files/configs (eg: path files)

Short Description

Possible solution

How it would help others

Whether your problem is related or not, I added the heartbeat to those tasks to avoir having zero activity during evolve.
However since you also get issues during transfer, you issue surely comes from something else.

Ok, seems i found source of a problem.
This is social_handler.py when client attempting to connect with mqtt_protocol version.

#5240

I get error 54 not 104 so I'm not sure of that issue is related to mine. But as your instructions I disables social and will report back to you if that helped. Thanks.

This error is completely ignorable. Most likely will be addressed during major social_handler.py overhaul (currently in progress).


InvalidArgumentError : restore the weight of Inception_v4

System information

OS Platform and Distribution : windows10
TensorFlow installed from : Source
TensorFlow version : 1.7.0
Python version: Python 3.6.3
Bazel version: 0.11.1
GCC/Compiler version : 5.4.0
CUDA/cuDNN version: no
GPU model and memory: no
Exact command to reproduce:

Describe the problem
I'm training a deep neural network.I want to restore the weight of Inception_v4 ,but there are some errors.I think the define of network is right but I can't restore the weight of model.Is it a bug of different version.
The errors:
InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [3,3,128,768] rhs shape= 5,5,128,768, use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](InceptionV4/AuxLogits/Conv2d_2a/weights, save/RestoreV2:9)]]
Source code
import scipy.misc as misc
import pickle
import tensorflow as tf
from tqdm import tqdm
import numpy as np
import argparse
import fnmatch
import sys
import os

from nets.inception_v4 import *
sys.path.insert(0, 'nets/')
slim = tf.contrib.slim

def getPaths(data_dir):
    image_paths = []
    # add more extensions if need be
    ps = ['jpg', 'jpeg', 'JPG', 'JPEG', 'bmp', 'BMP', 'png', 'PNG']
    for p in ps:
        pattern = '*.' + p
        for d, s, fList in os.walk(data_dir):
            for filename in fList:
                if fnmatch.fnmatch(filename, pattern):
                    fname_ = os.path.join(d, filename)
                    image_paths.append(fname_)
    return image_paths

if name == 'main':
    parser = argparse.ArgumentParser()
    parser.add_argument('--data_dir', required=True, type=str, help='Directory images are in. Searches recursively.')
    parser.add_argument('--model', required=True, type=str, help='Model to use')
    parser.add_argument('--checkpoint_file', required=True, type=str, help='Model file')
    a = parser.parse_args()
    data_dir = a.data_dir
    model = a.model
    checkpoint_file = a.checkpoint_file
    # I only have these because I thought some take in size of (299,299), but maybe not
    height, width, channels = 224, 224, 3
    x = tf.placeholder(tf.float32, shape=(1, height, width, channels))

    arg_scope = inception_v4_arg_scope()
    with slim.arg_scope(arg_scope):
        logits, end_points = inception_v4(x, is_training=False, num_classes=1001)
        features = end_points['PreLogitsFlatten']

    sess = tf.Session()
    saver = tf.train.Saver()
    saver.restore(sess, checkpoint_file)

    feat_dict = {}
    paths = getPaths(data_dir)
    print('Computing features...')
    for path in tqdm(paths):
        image = misc.imread(path)
        image = misc.imresize(image, (height, width))
        image = np.expand_dims(image, 0)
        feat = np.squeeze(sess.run(features, feed_dict={x: image}))
        feat_dict[path] = feat

    try:
        os.makedirs('features/')
    except:
        pass
    exp_pkl = open('features/' + model + '_features.pkl', 'wb')
    data = pickle.dumps(feat_dict)
    exp_pkl.write(data)
    exp_pkl.close()

Log：
D:\PythonProject\Compute-Features-WithModel>python compute_features.py --data_dir=jaffe/ --checkpoint_file=inception_v4.ckpt --model=inception_v4
Use the retry module or similar alternatives.
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1327, in _do_call
    return fn(*args)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1312, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1420, in _call_tf_sessionrun
    status, run_metadata)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 516, in exit
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [3,3,128,768] rhs shape= [5,5,128,768]
		[[Node: save/Assign_9 = Assign[T=DT_FLOAT, _class=["loc:@InceptionV4/AuxLogits/Conv2d_2a/weights"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](InceptionV4/AuxLogits/Conv2d_2a/weights, save/RestoreV2:9)]]

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "compute_features.py", line 120, in <module>
    saver.restore(sess, checkpoint_file)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\training\saver.py", line 1775, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 905, in run
    run_metadata_ptr)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1140, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1321, in _do_run
    run_metadata)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1340, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [3,3,128,768] rhs shape= [5,5,128,768]
         [[Node: save/Assign_9 = Assign[T=DT_FLOAT, _class=["loc:@InceptionV4/AuxLogits/Conv2d_2a/weights"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](InceptionV4/AuxLogits/Conv2d_2a/weights, save/RestoreV2:9)]]
Caused by op 'save/Assign_9', defined at:
  File "compute_features.py", line 119, in <module>
    saver = tf.train.Saver()
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\training\saver.py", line 1311, in init
    self.build()
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\training\saver.py", line 1320, in build
    self.build(self.filename, build_save=True, build_restore=True)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\training\saver.py", line 1357, in _build
    build_save=build_save, build_restore=build_restore)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\training\saver.py", line 809, in _build_internal
    restore_sequentially, reshape)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\training\saver.py", line 470, in _AddRestoreOps
    assign_ops.append(saveable.restore(saveable_tensors, shapes))
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\training\saver.py", line 162, in restore
    self.op.get_shape().is_fully_defined())
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\ops\state_ops.py", line 281, in assign
    validate_shape=validate_shape)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\ops\gen_state_ops.py", line 64, in assign
    use_locking=use_locking, name=name)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 3290, in create_opop_def=op_def)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 1654, in init
    self.traceback = self.graph._extract_stack()  # pylint: disable=protected-access
InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [3,3,128,768] rhs shape= [5,5,128,768]
         [[Node: save/Assign_9 = Assign[T=DT_FLOAT, _class=["loc:@InceptionV4/AuxLogits/Conv2d_2a/weights"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](InceptionV4/AuxLogits/Conv2d_2a/weights, save/RestoreV2:9)]]

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce

I have added this information。

This question is better asked on StackOverflow since it is not a bug or feature request. There is also a larger community that reads questions there.
If you think we've misinterpreted a bug, please comment again with a clear explanation, as well as all of the information requested in the issue template. Thanks!


TST: optimize.linprog/milp: add tests for various bug reports

Reference issue
Closes gh-18907  # already fixed by HiGHS 1.6.0, it seems
Closes gh-20116
Closes gh-20336
What does this implement/fix?
There were a few open bug reports that will may be closed after upgrading to the most recent HiGHS. In case we want regression tests, here they are. If not (since these are all upstream fixes), consider this a meta-issue of bugs for which we're looking for upstream fixes.

I'm +1 on adding these in. Having tests gives me a greater sense of security that the software we use is actually working

Waiting until the new HiGHs gets in. I don't think these tests all pass on main.

Looks like tests were added in gh-21565. After CI finishes, I'll document what happened, and update the linked issues, and close the PR.

This does not close gh-20116 and gh-20336; those have been reported to the HiGHS developers.
@HaoZeke based on the previous test run, it looks like the test for gh-18907 was passing even on 32-bit platforms. If everything comes back green here, any reason why we shouldn't merge this and close gh-18907?

If I hear back otherwise, we can revert; otherwise, self-merging since this just uncomments a test that seems to be passing now.


This page is also linked from 'Built-in components > Title'

Under the list of built-in components is a link for "Title". But it links to https://docs.microsoft.com/en-us/aspnet/core/blazor/fundamentals/additional-scenarios which appears to be an unrelated page which has a ToC entry under 'Fundamentals > Additional Scenarios'.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: fc56d254-09c5-993b-2559-76af9a3f97ed
Version Independent ID: a0bb183f-7e4d-ba39-c94c-cf60e951c7a9
Content: ASP.NET Core Blazor hosting model configuration
Content Source: aspnetcore/blazor/fundamentals/additional-scenarios.md
Product: aspnet-core
Technology: aspnetcore-blazor
GitHub Login: @guardrex
Microsoft Alias: riande

Thanks for the report @iadaz ... I'll take a look.

@iadaz ... This is a point-in-time problem before we reach .NET 5 GA. We don't have a way to version the sidebar ToC. Therefore, a link there for .NET 5 preview content will appear to go nowhere 😨. If you flip that doc over to the .NET 5 preview version, you'll get the content ...
https://docs.microsoft.com/en-us/aspnet/core/blazor/fundamentals/additional-scenarios?view=aspnetcore-5.0#influence-html-head-tag-elements
I've thought about placing a ToC update in to address the entries where each .NET 5 feature would have something like "(.NET 5 Preview)" after the entry; but we're so close to .NET 5 GA, that I think we can probably leave it. Although I think we can wing it this time, I think next time (5.1? ... 6.0 if I'm still around) we should have a cue like that on the entries.

🤔 ...... I'll keep an 👂 open for more feedback. If more devs hit this, then I will go ahead and put that cue on the entries for .NET 5. Let's see how it goes.
Thanks again for writing in on it.

btw @scottaddie ... I suppose that another way to go would be to specify the version of the loaded doc with the query string param ...
?view=aspnetcore-5.0

... my concern there is that we would be switching the version without the reader knowing/noticing ... we may generate a worse problem than we currently have.
I think for now let's keep going. We're getting close to release. I'll just let devs know what to do when this happens for now.


add example:docker moderate-custom-network-bridge-driver

I've completed task 12: create a new bridge network with custom IP address moderate-custom-network-bridge-driver

It has a logical network problem. I will solve it.
Please check your code for the next commits. Thanks.


Provide uninstall instructions in the README

It might be useful to demonstrate that rbenv doesn't keep your system hostage, and that it's easy to disable or uninstall.
Is this too wordy?
Fixes #746
/cc @jchristi @sstephenson

Looks great.

Wow, amazing turnaround time! :+1:


Batch size changes inference

Issue should be reproducible with this notebook (EDIT: updated link, previous link was made editable by anyone).
(rest is copied from this issue but they can't help with decapoda-research/llama-7b-hf which I assume alpaca-lora is for)
Basically when I run generate with just one tokenized sequence, I get a certain result, and when I process the same sequence but inside a batch instead, the result changes. To make sure it wasn't any tokenization shenanigans, I tokenized the whole batch and took just the parts that related to the weird sequence (basically just {k: v[1:] for k, v in tokenized_inputs.items()} where the batch is just two items and the weird one is the second item).
For clarity, when I pass just
"""Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
Combine the question and answer into an image caption as succinctly as possible. Be sure to include the phrase "a photo of". Do not draw false conclusions.

### Input:
Is this a baseball game? no

### Response:
"""

to the generate function of a LlamaForCausalLM wrapped in a PeftModel, I get
This is not a baseball game.

However, if I put it in a two-item batch, the output for some reason is instead
A photo of people playing a game.

Just reproduced  your code,
There are two issues:

Instruction format should be:
instruction = "Combine the question and answer into an image caption as succinctly as possible. Be sure to include the phrase a photo of. Do not draw false conclusions."
need to use bach_encode before bach_decode:
tokenized_inputs = tokenizer.batch_encode_plus(prompts, padding=True, truncation=False,return_tensors='pt').to('cuda')

Then the output would be:
<unk>Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
Combine the question and answer into an image caption as succinctly as possible. Be sure to include the phrase a photo of. Do not draw false conclusions.

### Input:
Is this a baseball game? no

### Response:
A photo of a baseball game.
--------------------------------------------------------------------------------
<unk>Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
Combine the question and answer into an image caption as succinctly as possible. Be sure to include the phrase a photo of. Do not draw false conclusions.

### Input:
Is this a baseball game? no

### Response:
A photo of a baseball game.

Batch size should change inference, no? Batch size is a method to aggregate and smooth the model's training, in an attempt to speed up the processing of massive changes to the gradient by processing more information at once.
A lower batch size will result in the model seeing more nuanced, piece-meal, local minima data, which should result in different inference.

@ElleLeonne I'm not training though; what I mean is that during inference, model outputs are changing depending on how many outputs are being generated at a time, which shouldn't be the case. It's like having an image classifier that can correctly classify an image, however once you classify the same image but within a batch it starts changing its predictions.
@lywinged Thanks, I'll look into this! Mind if I ask what the problem with the original instruction is? Are there issues with using quotes?

Hi @lywinged I don't think the suggestions solve the underlying issue. If I follow both of them, different outputs are still being produced on inputs = ["What country's flag is stitched to the back of the jeans? England", "What country's flag is stitched to the back of the jeans? usa"].
Btw doesn't a tokenizer's __call__ function just call batch_encode_plus anyway if you pass in a list of strings?

Hi @lywinged I don't think the suggestions solve the underlying issue. If I follow both of them, different outputs are still being produced on inputs = ["What country's flag is stitched to the back of the jeans? England", "What country's flag is stitched to the back of the jeans? usa"].
Btw doesn't a tokenizer's __call__ function just call batch_encode_plus anyway if you pass in a list of strings?

https://colab.research.google.com/drive/1mBSGHmfZqb5MaFDaYfbNq5CedrBjEJb3?usp=share_link

Thanks, though the suggested changes still don't fix the underlying problem I think. Like I mentioned here

Hi @lywinged I don't think the suggestions solve the underlying issue. If I follow both of them, different outputs are still being produced on inputs = ["What country's flag is stitched to the back of the jeans? England", "What country's flag is stitched to the back of the jeans? usa"].
Btw doesn't a tokenizer's __call__ function just call batch_encode_plus anyway if you pass in a list of strings?

you can change the inputs and get the same issue of "changing batch size during inference changes the generated output for some reason".
https://colab.research.google.com/drive/1AAk5uxLjdkmXDAJRocGul5eyucSaoS01?usp=sharing

Thanks, though the suggested changes still don't fix the underlying problem I think. Like I mentioned here

Hi @lywinged I don't think the suggestions solve the underlying issue. If I follow both of them, different outputs are still being produced on inputs = ["What country's flag is stitched to the back of the jeans? England", "What country's flag is stitched to the back of the jeans? usa"].
Btw doesn't a tokenizer's __call__ function just call batch_encode_plus anyway if you pass in a list of strings?

you can change the inputs and get the same issue of "changing batch size during inference changes the generated output for some reason".
https://colab.research.google.com/drive/1AAk5uxLjdkmXDAJRocGul5eyucSaoS01?usp=sharing

print(tokenizer.batch_decode(full_output, skip_special_tokens=False)[0])
print('-' * 80)
print(tokenizer.batch_decode(full_output, skip_special_tokens=False)[1])

<unk><unk>Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
Combine the question and answer into an image caption as succinctly as possible. Be sure to include the phrase a photo of. Do not draw false conclusions.

### Input:
What country's flag is stitched to the back of the jeans? England

### Response:
A photo of England's flag stitched to the back of the jeans.
--------------------------------------------------------------------------------
<unk> Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
Combine the question and answer into an image caption as succinctly as possible. Be sure to include the phrase a photo of. Do not draw false conclusions.

### Input:
What country's flag is stitched to the back of the jeans? usa

### Response:
A photo of the USA flag stitched to the back of the jeans.<unk>

Sorry I'm not entirely sure what this code snippet

print(tokenizer.batch_decode(full_output, skip_special_tokens=False)[0])
print('-' * 80)
print(tokenizer.batch_decode(full_output, skip_special_tokens=False)[1])

you shared is for. We're still getting different outputs for tokenizer.batch_decode(full_output, skip_special_tokens=False)[1] and tokenizer.batch_decode(single_output, skip_special_tokens=False)[0], which is the problem.

Sorry I'm not entirely sure what this code snippet

print(tokenizer.batch_decode(full_output, skip_special_tokens=False)[0])
print('-' * 80)
print(tokenizer.batch_decode(full_output, skip_special_tokens=False)[1])

you shared is for. We're still getting different outputs for tokenizer.batch_decode(full_output, skip_special_tokens=False)[1] and tokenizer.batch_decode(single_output, skip_special_tokens=False)[0], which is the problem.

In fact, your question has nothing to do with alpaca-lora, that's basic skill issue, but share your notebook to me and make anyone editable.

Sure, here's the link to the original notebook, now editable.
Again, I'm not entirely sure what's happening here. Giving the same input to the model (with should have do_sample=False by default) yields different outputs depending on whether or not the input is within a batch with other inputs. Even with the suggested changes, this issue still arises with inputs = ["What country's flag is stitched to the back of the jeans? England", "What country's flag is stitched to the back of the jeans? usa"]. I'm not necessarily looking for "correct" answers (both of the answers I got for this input made sense) but the fact that it changes on the exact same input without sampling implies something is wrong.

Sure, here's the link to the original notebook, now editable.
Again, I'm not entirely sure what's happening here. Giving the same input to the model (with should have do_sample=False by default) yields different outputs depending on whether or not the input is within a batch with other inputs. Even with the suggested changes, this issue still arises with inputs = ["What country's flag is stitched to the back of the jeans? England", "What country's flag is stitched to the back of the jeans? usa"]. I'm not necessarily looking for "correct" answers (both of the answers I got for this input made sense) but the fact that it changes on the exact same input without sampling implies something is wrong.

It's done. Please check the notebook.

Unfortunately I can't test with yahma/llama-7b-hf because the shard sizes are too big for my runtime. Would you mind running this edited version of your last code cell?:
def generate_prompt(instruction, input):
    return f"Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\n{instruction}\n\n### Input:\n{input}\n\n### Response:\n"

instruction_old = 'Combine the question and answer into an image caption as succinctly as possible. Be sure to include the phrase "a photo of". Do not draw false conclusions.'
instruction = "Combine the question and answer into an image caption as succinctly as possible. Be sure to include the phrase a photo of. Do not draw false conclusions."
inputs = ["What country's flag is stitched to the back of the jeans? England", "What country's flag is stitched to the back of the jeans? usa"]
prompts_old = [generate_prompt(instruction_old, inp) for inp in inputs]
print(prompts_old)
prompts = [generate_prompt(instruction, inp) for inp in inputs]
print(prompts)

tokenized_inputs = tokenizer(prompts, padding=True, truncation=True, return_tensors="pt").to('cuda')
tokenized_inputs_old = tokenizer(prompts_old, padding=True, truncation=True, return_tensors="pt").to('cuda')
print(tokenized_inputs_old)
print(tokenized_inputs)

generation_config = GenerationConfig(max_new_tokens=128)

#use old instruction with double quotation marks inside:
with torch.no_grad():
    full_output = model.generate(
        **tokenized_inputs_old,
        generation_config=generation_config
    )

    # same tokenized input, but we index just into the parts for the second input
    single_output = model.generate(
        **{k: v[1:] for k, v in tokenized_inputs_old.items()},
        generation_config=generation_config
    )

print(tokenizer.batch_decode(full_output, skip_special_tokens=False))
print('-' * 80)
print(tokenizer.batch_decode(full_output, skip_special_tokens=False)[-1])
print('-' * 80)
print(tokenizer.batch_decode(single_output, skip_special_tokens=False)[0])

I just want to see if the issue persists with other inputs even with an updated tokenizer.

Unfortunately I can't test with yahma/llama-7b-hf because the shard sizes are too big for my runtime. Would you mind running this edited version of your last code cell?:
def generate_prompt(instruction, input):
    return f"Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\n{instruction}\n\n### Input:\n{input}\n\n### Response:\n"

instruction_old = 'Combine the question and answer into an image caption as succinctly as possible. Be sure to include the phrase "a photo of". Do not draw false conclusions.'
instruction = "Combine the question and answer into an image caption as succinctly as possible. Be sure to include the phrase a photo of. Do not draw false conclusions."
inputs = ["What country's flag is stitched to the back of the jeans? England", "What country's flag is stitched to the back of the jeans? usa"]
prompts_old = [generate_prompt(instruction_old, inp) for inp in inputs]
print(prompts_old)
prompts = [generate_prompt(instruction, inp) for inp in inputs]
print(prompts)

tokenized_inputs = tokenizer(prompts, padding=True, truncation=True, return_tensors="pt").to('cuda')
tokenized_inputs_old = tokenizer(prompts_old, padding=True, truncation=True, return_tensors="pt").to('cuda')
print(tokenized_inputs_old)
print(tokenized_inputs)

generation_config = GenerationConfig(max_new_tokens=128)

#use old instruction with double quotation marks inside:
with torch.no_grad():
    full_output = model.generate(
        **tokenized_inputs_old,
        generation_config=generation_config
    )

    # same tokenized input, but we index just into the parts for the second input
    single_output = model.generate(
        **{k: v[1:] for k, v in tokenized_inputs_old.items()},
        generation_config=generation_config
    )

print(tokenizer.batch_decode(full_output, skip_special_tokens=False))
print('-' * 80)
print(tokenizer.batch_decode(full_output, skip_special_tokens=False)[-1])
print('-' * 80)
print(tokenizer.batch_decode(single_output, skip_special_tokens=False)[0])

I just want to see if the issue persists with other inputs even with an updated tokenizer.

[' Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nCombine the question and answer into an image caption as succinctly as possible. Be sure to include the phrase "a photo of". Do not draw false conclusions.\n\n### Input:\nWhat country's flag is stitched to the back of the jeans? England\n\n### Response:\nA photo of England's flag stitched to the back of the jeans.', ' Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nCombine the question and answer into an image caption as succinctly as possible. Be sure to include the phrase "a photo of". Do not draw false conclusions.\n\n### Input:\nWhat country's flag is stitched to the back of the jeans? usa\n\n### Response:\nA photo of the USA flag stitched to the back of the jeans.']
 Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
Instruction:
Combine the question and answer into an image caption as succinctly as possible. Be sure to include the phrase "a photo of". Do not draw false conclusions.
Input:
What country's flag is stitched to the back of the jeans? usa
Response:
A photo of the USA flag stitched to the back of the jeans.
 Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
Instruction:
Combine the question and answer into an image caption as succinctly as possible. Be sure to include the phrase "a photo of". Do not draw false conclusions.
Input:
What country's flag is stitched to the back of the jeans? usa
Response:
A photo of the USA flag stitched to the back of the jeans.
I suggest to use cloud server, it's only $0.1 per hour for this small stuff.

Thanks! In that case looks like it was issues with the tokenizer. Still a little confused over how that's possible given that 1) the tokenization was the same across both batched and non-batched inference tests anyway and 2) do_sample was set to False, but the important thing is that the issue is sort of fixed. Thanks again!


bug fix search package for multiple sources

https://github.com/aliasadidev/vsocde-npm-gui/issues/9

Hi @TomyCesaille.
Thank you for your collaboration.
I think we have an issue. If we've two repositories and both have the same result, it's not a good idea that concat them together (duplicate results may occur).
I think it's better that, Insert all results into one list, and makes it unique, and sort it.
I've prepared a sample code.
let server1Result = [
    "Microsoft.Extensions.DependencyInjection",
    "Microsoft.Extensions.Configuration.Json",
    "Microsoft.AspNetCore.TestHost",
    "Microsoft.NET.Test.Sdk"
];
let server2Result = [
    "Microsoft.Extensions.DependencyInjection",
    "Microsoft.Extensions.Configuration.Json",
    "Microsoft.AspNetCore.TestHost",
    "Microsoft.NET.Test.Sdk",
    "Microsoft.EntityFrameworkCore.Relational",
    "Microsoft.EntityFrameworkCore.Abstractions"
];
let server3Result = [
    "Microsoft.Extensions.DependencyInjection",
    "Microsoft.Extensions.Configuration.Json",
    "Microsoft.NET.Test.Sdk2",
];
/// serverNResult ...

let keyValuePackages: Record<string, number> = {};
let packages = [...server1Result, ...server2Result, ...server3Result];
packages.forEach(pkg => {
    keyValuePackages[pkg] = keyValuePackages[pkg] ? keyValuePackages[pkg] + 1 : 1;
});
let resultConsolided = Object.keys(keyValuePackages).sort((n1, n2) => keyValuePackages[n2] - keyValuePackages[n1]);
console.log(resultConsolided)

Hi @TomyCesaille.
Thank you for your collaboration.
I think we have an issue. If we've two repositories and both have the same result, it's not a good idea that concat them together (duplicate results may occur).
I think it's better that, Insert all results into one list, and makes it unique, and sort it.
I've prepared a sample code.
let server1Result = [
    "Microsoft.Extensions.DependencyInjection",
    "Microsoft.Extensions.Configuration.Json",
    "Microsoft.AspNetCore.TestHost",
    "Microsoft.NET.Test.Sdk"
];
let server2Result = [
    "Microsoft.Extensions.DependencyInjection",
    "Microsoft.Extensions.Configuration.Json",
    "Microsoft.AspNetCore.TestHost",
    "Microsoft.NET.Test.Sdk",
    "Microsoft.EntityFrameworkCore.Relational",
    "Microsoft.EntityFrameworkCore.Abstractions"
];
let server3Result = [
    "Microsoft.Extensions.DependencyInjection",
    "Microsoft.Extensions.Configuration.Json",
    "Microsoft.NET.Test.Sdk2",
];
/// serverNResult ...

let keyValuePackages: Record<string, number> = {};
let packages = [...server1Result, ...server2Result, ...server3Result];
packages.forEach(pkg => {
    keyValuePackages[pkg] = keyValuePackages[pkg] ? keyValuePackages[pkg] + 1 : 1;
});
let resultConsolided = Object.keys(keyValuePackages).sort((n1, n2) => keyValuePackages[n2] - keyValuePackages[n1]);
console.log(resultConsolided)

I don't know how VS2019-17 handle the duplicates though. I've been avoiding this risks for years.
Thanks for the code sample, I'll integrate it.
First source listed will have priority in case of duplicates; I guess this is fair.

I don't know how VS2019-17 handle the duplicates though. I've been avoiding this risks for years.
Thanks for the code sample, I'll integrate it.
First source listed will have priority in case of duplicates; I guess this is fair.

New commits gives:

No duplicates (first source will be kept)
Best matching algorithm as follow:

const isBestMatch = (packageNuget: any, query: string) => {
    return packageNuget.id.toLowerCase().startsWith(query.toLowerCase());
};

let packagesUniques = uniqBy(packages, "id");
let packagesBestMatch = packagesUniques.filter(o => isBestMatch(o, query));
packagesBestMatch = packagesBestMatch.concat().sort(sortBy("id"));
let packagesOthers = packagesUniques.filter(o => !isBestMatch(o, query));
packagesOthers = packagesOthers.concat().sort(sortBy("id"));

return { data: [...packagesBestMatch, ...packagesOthers], totalHits: packagesUniques.length };

It returns the package that match the best based on the startsWith logic. It mimic quite well the way VS 2017-19 rank results with multiple sources (they maybe use the downloaded counter in consideration ?).
Counting the number of duplicates between sources and use it a a sort filter like you were suggestion in your code sample is not really relevant for user experience.
Looks good to you ?
Also, I specified your .ts formatter in .vscode/settings.json for the futur collaborators.

New commits gives:

No duplicates (first source will be kept)
Best matching algorithm as follow:

const isBestMatch = (packageNuget: any, query: string) => {
    return packageNuget.id.toLowerCase().startsWith(query.toLowerCase());
};

let packagesUniques = uniqBy(packages, "id");
let packagesBestMatch = packagesUniques.filter(o => isBestMatch(o, query));
packagesBestMatch = packagesBestMatch.concat().sort(sortBy("id"));
let packagesOthers = packagesUniques.filter(o => !isBestMatch(o, query));
packagesOthers = packagesOthers.concat().sort(sortBy("id"));

return { data: [...packagesBestMatch, ...packagesOthers], totalHits: packagesUniques.length };

It returns the package that match the best based on the startsWith logic. It mimic quite well the way VS 2017-19 rank results with multiple sources (they maybe use the downloaded counter in consideration ?).
Counting the number of duplicates between sources and use it a a sort filter like you were suggestion in your code sample is not really relevant for user experience.
Looks good to you ?
Also, I specified your .ts formatter in .vscode/settings.json for the futur collaborators.

The sort has a problem in some cases (When the popular packages don't start with the user query).
The Nuget result

The extension result

I think using the totalDownloads property is more efficient.

The sort has a problem in some cases (When the popular packages don't start with the user query).
The Nuget result

The extension result

I think using the totalDownloads property is more efficient.

Sorting by installs was my first try. I didn't liked the behavior so much.
For instance, I have a Some.MongoWrapper NuGet in my company NuGet source.
Searching Some.Mongo is ranking Some.MongoWrapper first in VS2017-19 (despite the poor 1k downloads).
However, sorting by downloads in here would shows the official MongoDB drivers packages first which can be annoying.

Sorting by installs was my first try. I didn't liked the behavior so much.
For instance, I have a Some.MongoWrapper NuGet in my company NuGet source.
Searching Some.Mongo is ranking Some.MongoWrapper first in VS2017-19 (despite the poor 1k downloads).
However, sorting by downloads in here would shows the official MongoDB drivers packages first which can be annoying.

We have two types of users:

Those users that use the single repository
Those users that use more than 1 repository

For type 1, we don't need to sort and unique it
For type 2, your solution would be better( the company's packages aren't always at the end of the list)
We have to see both of them

We have two types of users:

Those users that use the single repository
Those users that use more than 1 repository

For type 1, we don't need to sort and unique it
For type 2, your solution would be better( the company's packages aren't always at the end of the list)
We have to see both of them

Nice implementation ;)

Nice implementation ;)

I waiting for PR

I waiting for PR


Rename ServerStartUpUtil to ServerStartupUtil

What changes were proposed in this pull request?
This pr rename ServerStartUpUtil to ServerStartupUtil.

How about move it to sql/service/src/main/java/org/apache/spark/sql/service/utils?


initial unittest project setup

a very basic setup for (unit) testing

.. not sure about the artwork :)

Okay. Please update the assets right away.
(use T10 assets in /Assets off the root)

Then re-submit.


Multiple servers and memcached sessions

Added ability to use multiple servers.  Multiple servers can come from php.ini if Memcached sessions are available by using the string 'session.save_path'.
Also fixed some potential session collisions when using memcached sessions and other software wich uses the $_SESSSION['username'] session variable.  Improved security by checking username is in usernames array.

This is old now, closing


Keep report from ever scrolling or having a fixed width on report page

Keep report from ever scrolling or having a fixed width on report page

Change last child right padding to zero
.report-table th:last-child{
padding-right: 0px
}
Get rid of y overflow on reports main
.reports-main {
width: 100%;
}


Feature/quantity types and units

Minimally addresses #111

Made the changes we discussed and it's ready to 🚀 @gedaskir


[Draft] Dotnet SDK changes for 2019-10-01 Tags API for resources and subscriptions

Update Generated code;
Add tag related test cases;
Update SessionRecords due to API version change.

Hi @Grayer123  please remember to remove the "[Draft]" or "WIP" or anything like that otherwise the review request might have been ignored...


Merge blog and activity page to one

Task

[x] Merge blog and activity page to blog page.

Well done.


Update v_generate_user_grant_revoke_ddl.sql

Issue #, if available:
#459
Description of changes:
Explicitly cast DDL column to varchar 65536 rather than rely on text cast which does not set proper precision to align with other unioned selects
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Hi, I will be adding a fix along with other fixes in the #471 to handle this. Using varchar(max) is not recommended since it can slow down the query. I have changed the width of the ddl column to varchar(4000) which should be sufficient for any use case.

Closing per fix in #471 by Adedotua


Travis CI: Temporarily disable caching

So that the build will stop failing due to https://github.com/travis-ci/travis-ci/issues/5092
Successor to #18352.
CC: @hnrch02 @XhmikosR

:+1:


Missing dependencies

I'm getting a wierd missing dependencies error when trying to load the game.
The files it lists seems to be mostly there however nothing really opens in godot editor...

Ok, I think it's fixed

Ok, I think it's fixed

Yessir. I just tested it. It works now!


Add SE-ResNet{50,101,152} and SE-ResNeXt{50,101}_32x4d

I have ported the remaining SE-Res{Net, NeXt} models from https://github.com/hujie-frank/SENet
Table with acc@1 and acc@5 values for the models (values for the original models were taken from https://github.com/hujie-frank/SENet#trained-models).

Model
Version
Acc@1
Acc@5

SE-ResNeXt101_32x4d
Our porting
80.236
95.028

SE-ResNeXt101_32x4d
Caffe
80.19
95.04

SE-ResNeXt50_32x4d
Our porting
79.076
94.434

SE-ResNeXt50_32x4d
Caffe
79.03
94.46

SE-ResNet152
Caffe
78.66
94.46

SE-ResNet152
Our porting
78.658
94.374

SE-ResNet101
Our porting
78.396
94.258

SE-ResNet101
Caffe
78.25
94.28

SE-ResNet50
Caffe
77.63
93.64

SE-ResNet50
Our porting
77.636
93.752

Weights (there are 5 files in the zip archive): https://drive.google.com/open?id=1gjJ0NY0iSZiyIB_PcmMi7jTdn85VgV-M
I have also commented some parts of the code to explain implementation.

Amazing! Thanks a lot for your contribution :)
I just uploaded and tested your new networks. It works well.

Why not convert the last network SE-BN-Inception also? Thanks