Handle the proto-copying with Go

People are having trouble generating the protos on Mac because cp behaves differently compared to Linux.  This is annoying, so I have moved most of the logic to Go; we still have a shell script to run protoc and gopatch, but we do everything else with a custom program.  We have also had trouble with GNU tar vs. BSD tar before, so again, I avoided trying to invoke tar in the shell script and just use archive/tar.
The actual proto generation now runs as a genrule, which means that it's cached when the protos don't change.
I added a "test" mode and a test to check that your working copy matches what came out of the genrule.  If it doesn't match, then run "extract" (still aliased to //:make_proto).  The test output, and the output of make proto, look something like this:
INFO prototar/main.go:187 updated file {"name": "proto-docs.json"}
INFO prototar/main.go:187 updated file {"name": "proto-docs.md"}
INFO prototar/main.go:187 updated file {"name": "src/internal/jsonschema/admin_v2/ClusterInfo.schema.json"}
INFO prototar/main.go:187 updated file {"name": "src/openapi/pachyderm_api.swagger.json"}
INFO prototar/main.go:187 updated file {"name": "src/typescript/admin/admin.pb.ts"}
INFO prototar/main.go:187 updated file {"name": "src/admin/admin.pb.go"}
INFO prototar/main.go:187 updated file {"name": "src/admin/admin.pb.validate.go"}
INFO prototar/main.go:187 updated file {"name": "src/admin/admin.pb.zap.go"}

*** It looks like you might need to regenerate the protos in your working copy.

It will also print files that are unchanged (in debug mode) and files that it's deleting (when you remove a message from a proto and the JSON schema file needs to be killed).
I added a feature that lets you see "forgotten" files that you might have forgotten to copy over because they were generated in a weird location.  Do something like bazel build //src/proto:gen_go_proto_bundle; cat bazel-bin/src/proto/go_protos_forgotten_files.txt to see them.  This looks something like:
out/github.com/chrusty/protoc-gen-jsonschema/json-schema-options.pb.go
out/github.com/chrusty/protoc-gen-jsonschema/json-schema-options.pb.validate.go
out/github.com/chrusty/protoc-gen-jsonschema/json-schema-options.pb.zap.go
out/github.com/envoyproxy/protoc-gen-validate/validate/validate.pb.go
out/github.com/envoyproxy/protoc-gen-validate/validate/validate.pb.validate.go
out/github.com/envoyproxy/protoc-gen-validate/validate/validate.pb.zap.go

Indeed, we don't want those.  But now we can be sure that nothing is being forgotten, which will be helpful when someone adds a weird plugin.

Works as expected on Mac 👍


SUP-2343: Add lists of Attributes that can be used with buildkite-agent step

Had a customer reach out checking which attributes could be used with the buildkite-agent step [get|update] commands as they found it unclear from the docs. So adding list of available Attributes for clarity.
Will need to also update the Agents docs - https://github.com/buildkite/agent/blob/d387c3bdd58eb3fd94906e3e400ca8e0ab230e87/clicommand/step_get.go#L26-L31 - to correct example as retry is not an available attribute at the time of writing.

Will update PR, if code changes in Agent are merged: https://github.com/buildkite/agent/pull/2879

Agent v3.75.0 released with changes to help output


support github ci and e2e test

What type of PR is this?

Uncomment only one  /kind <> line, hit enter to put that in a new line, and remove leading whitespace from that line:
/kind bug
/kind documentation
/kind enhancement
/kind good-first-issue
/kind feature
/kind question
/kind design
/sig ai
/sig iot
/sig network
/sig storage
/sig storage

What this PR does / why we need it:
Add e2e test and github ci
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:

Does this PR introduce a user-facing change?

other Note

/assign @kadisi

@huiwq1990 please fix the conflicting files in ci pipeline

close, new pr: https://github.com/openyurtio/yurt-app-manager/pull/107


"Sorting Using Directives" is missing

Environment data
dotnet --info output:
.NET Command Line Tools (1.0.0-rc4-004771)
Product Information:
Version:            1.0.0-rc4-004771
Commit SHA-1 hash:  4228198f0e
Runtime Environment:
OS Name:     Mac OS X
OS Version:  10.11
OS Platform: Darwin
RID:         osx.10.11-x64
Base Path:   /usr/local/share/dotnet/sdk/1.0.0-rc4-004771
VS Code version: 1.9.1
C# Extension version: 1.7.0
Steps to reproduce
Create a C# file in .NET Core project, add some using directives at the top of the file in an unsorted order.
Expected  behavior
It should provide an option to sort these using directives. (the old plugin for DNX provides this feature)
Actual behavior
No such feature available now.

We're tracking Sort Usings with https://github.com/OmniSharp/omnisharp-vscode/issues/788.


Exclusive item boosts (Merged)

Description:
I took the item boosts pr from @notmeta (#1409) and merged master into it, because there were merge conflicts
Other checks:

I don't know how to test, so I haven't tested it

I've now tested the code on my own test bot. Seems to be fully working.
The only boost added in this update is a 7% pegasians boost for zilyana instead of 5% range boost. Showcasing the exclusive item boost as well. Credit to @notmeta


Property getter and setter generated without method name

Discovered in my project: https://github.com/laggykiller/apngasm-python/tree/stubgen-fix
To build and reproduce
git clone --recursive https://github.com/laggykiller/apngasm-python.git
cd apngasm-python
git checkout stubgen-fix
python -m build .

https://github.com/laggykiller/apngasm-python/blob/stubgen-fix/src/apngasm_python.cpp#L239
        .def_prop_rw("width",
        [](apngasm::APNGFrame &t) APNGASM_PY_DECLSPEC { return t._width; },
        [](apngasm::APNGFrame &t, unsigned int v) APNGASM_PY_DECLSPEC { t.width(v); },
        R"pbdoc(
        The width of frame.
        )pbdoc")

The generated pyi
    @property
    def (self) -> int:
        """
                The width of frame.
        """
        ...
    @width.setter
    def (self, arg: int, /) -> None:
        """
                The width of frame.
        """
        ...

The expected correct pyi generated should contain
    @property
    def width(self) -> int:
        """
                The width of frame.
        """
        ...
    @width.setter
    def width(self, arg: int, /) -> None:
        """
                The width of frame.
        """
        ...

This should be fixed now and I am going to release a new version. Btw. the command python -m build . does not work since there is now build module available. Maybe there is an instruction missing? Anyway, it is much simpler to just install the pip package and call nanobind-stubgen apngasm_python._apngasm_python.
Btw. I also fixed the ndarray signatures.

Thank you for your hardwork. Can confirm the problem is fixed!

@cansik regarding ndarray signatures, would it be possible to automatically add import numpy.typing in the generated pyi file?

I guess a new issue would be great, otherwise I am also open for PR's anytime.
But regarding your questions:
'list' is parsed from nanobind docstring, but we can add a postprocessor of course
Imports are currently not handled well, it is more a guessing. I think it's necessary to add a proper import handling which detects modules and imports automatically.

@cansik opened a PR for that, plus fixing indentation in docstring


Invalid value for 'boolean' type in nested request body on apiary.io

The following gives me a warning for the subscription attribute, but not for the testBool attribute, and I have no idea why
## Collection [/user]

### Update user profile [PATCH /user/{id}]

+ Parameters
    + id (string) - Id of the user

+ Request

    + Attributes (object)
        + testBool (boolean)
        + arr (array[object])
            + id (string)
            + subscription (boolean)

    + Body

            {
            }

+ Response 204

The syntax for placing items in array is as follows:
+ Attributes (array)
    + hello (string)
    + true (boolean)
    + (object)
        + key: value

Which equates to the following structure in JSON:
[
  "hello",
  true
]

The line from your API Blueprint + subscription (boolean) is stating that you want a boolean, with the value "subscription" inside the array. As "subscription" is not a valid boolean (it is not true or false, the warning is emitted).
Instead, I think you are intending to wrap these array values in an object, for example an object with a key id and another with subscription. To do this in MSON you would write the following:
+ Attributes
    + arr (array[object])
        + (object)
            + id (string)
            + subscription (boolean)


About Us - Page

Page with description of what Code 4 Puerto Rico's mission and other details about the team.

A placeholder page has been deployed. We now need to add the proper content and layout. The deployed placeholder can be seen at https://peaceful-ardinghelli-f8d784.netlify.app/about/. I'll be keeping this issue open while we start constructing the layout and content.

Looking good, guys! Question: has any designer offered to do the styling/branding? I'd like to do it if no one else has. I had been doing a bit of research of comparable sites, and I'd like to give it a look & feel similar to these benchmarks (assuming the rest of the group likes the style and we'd make design options as a group).

@m3orange we are precisely at the stage where we have setup the project and have a way of creating multiple page. Now we need help from brigade designers for guidance on our layout and other aspects.
These screenshots look great! We're in a good place to implement them.
Feel free to kick off any wireframes and mocks and we can go through them as a group on our next Hack Night on 8/10.
We are using Tailwind CSS as our styling engine. It has a ton of utility CSS classes to make rapid prototyping easy. Everything is all set in the repository, all you have to do to apply these is to call the classes on the documentation.
https://tailwindcss.com/docs/container
Let me know if you have any other questions. I'm also available on our slack #website channel.

@m3orange Any help you can provide is more than welcomed.


Simplify - reduces nested array complexity to depth 1

_.simplify([[1, 2, 3], [2, 3, 4], [[[3, 4, 5], [6, 7, 8]]]])
=> [[1, 2, 3], [2, 3, 4], [3, 4, 5], [6, 7, 8]]

That's what _.flatten does.

Flatten does something different:
.flatten([[1,2,3], [2,3,4], [[[3,4,5], [6,7,8]]]])
=> [1, 2, 3, 2, 3, 4, 3, 4, 5, 6, 7, 8]
.flatten([[1,2,3], [2,3,4], [[[3,4,5], [6,7,8]]]], true)
=> [1, 2, 3, 2, 3, 4, Array[2]]
On Thu, Aug 6, 2015 at 6:38 PM, John-David Dalton<EMAIL_ADDRESS>wrote:

That's what _.flatten does.
—
Reply to this email directly or view it on GitHub
https://github.com/jashkenas/underscore/pull/2267#issuecomment-128530542
.

Flatten with shallow flag does this:
_.flatten([[1,2,3], [2,3,4], [[[3,4,5], [6,7,8]]]], true)
[1, 2, 3, 2, 3, 4, Array[2]]
Simplify does this:
_.simplify([[1,2,3], [2,3,4], [[[3,4,5], [6,7,8]]]])
[Array[3], Array[3], Array[3], Array[3]]
What do you mean by "mixin"?
Sorry, I'm a newbie.

Flatten does something different:

Correct. But it's close enough to where you could create something similar mapping over the array.

What do you mean by "mixin"?

My mixin I mean as a separate chunk of code that you _.mixin to Underscore.

Hmmm... Okay.
I just thought it would be super useful when used with .apply . Sometimes
.apply can be tricky because you have to do .apply(null, [a].concat(b)
where b is a nested array. I thought it would be easier to do .apply(null,
_.simplify(a, b, c))
I don't suppose you could leave it up for discussion?
If not, I understand.
Either way, thank you for your time.
On Thu, Aug 6, 2015 at 6:46 PM, John-David Dalton<EMAIL_ADDRESS>wrote:

Flatten does something different:
Correct. But it's close enough to where you could create something similar
mapping over the array.
What do you mean by "mixin"?
My mixin I mean as a separate chunk of code that you _.mixin
http://underscorejs.org/#mixin to Underscore.
—
Reply to this email directly or view it on GitHub
https://github.com/jashkenas/underscore/pull/2267#issuecomment-128532079
.


Use PORT0 as listen port if running in marathon

Use PORT0 as listen port if running in marathon.
It's very useful if bridge network is disabled.

PORT0 is offered in marathon environment，it's reasonable to use PORT0 as listen port in both run with HOST and BRIDGE network.

Hey @collinmsn thanks for the pr!
could you elaborate "It's very useful if bridge network is disabled."
Am I correct to say that with this change the current marathon file wouldn't work as the server will be listening on PORT0 which will be the same port than the assigned to the host but this one will be connecting to 5000 inside the container.
        {
          "containerPort": 5000,
          "hostPort": 0,
          "servicePort": 5000,
          "protocol": "tcp"
        }

Sorry that this pr will broken the marathon config. But if container port is not important, we can set containerPort to 0, which will get the same port as the hostPort. Then both bridge and host network will be OK.

This seems importan, to make it as dynamic as possible

I mean if it's not required to use fixed container port, we can use dynamic port offered by marathon. That's what this commit does.

Thanks, @collinmsn if you are using this ui in your company would you mind to add ir to the list in the Readme?
cheers


Deal with additional conditions properly

These changes are a fix for Issue https://github.com/cloudstateio/cloudstate/issues/536.
The fundamental diffs are

Make the Status component optional in both StatefulStores and StatefulServices.  This is the natural condition when they're first created.
Fix up ReconcileStatefulServiceConditions()

After these changes, and testing with the setup of Issue https://github.com/cloudstateio/cloudstate/issues/536, the StatefulService entity ends up with a Status section looking like:
status:
  conditions:
  - lastTransitionTime: "2021-02-18T01:15:21Z"
    message: Stateful store no-store not supported
    reason: StoreNotSupported
    status: "True"
    type: NotReady
  - lastTransitionTime: "2021-02-18T01:25:22Z"
    message: Application is in the process of updating
    reason: UpdateInProgress
    status: "True"
    type: NotReady

Tests now pass but converting to draft because I want to reconsider making the Status component optional.


add deleteFactor

initVairable test is failing in this branch and in master, but the new deleteFactor is working fine.

we should remember to add a testing for deleteFactor


Missing future starlink missions

November 22 starling 15 mission is not showing up or any other planned/ TBD missions for starlink. https://en.m.wikipedia.org/wiki/Starlink

Updated


Add internal/trace from golang

Currently adding that package to internal/trace, as I feel we shouldn't be exposing others author's packages for them. But if maintainers think it's more reasonable to make it non internal, then I can change the PR :)
The only thing I actually added is README.md, so it makes sense to only review that :)
Ref https://github.com/conprof/conprof/issues/46

It would be awesome to get this reviewed/merged as a lot of other PRs that I would like to open depends on this.

Could we add some automation on updating this? Otherwise stuff like this tends to get fragile quickly.

:+1: I'll do something like make sync-trace which git pull go repo and put it into right place

sounds good to me

Done :)


Allow file postprocessors like Prettier

Thanks for this module! One important feature that's missing to me is the ability to easily format generated files with some kind of postprocessor (eg. Prettier). For example, in the project I'm working on now, unformatted files will fail the build on CI.
This PR adds a simple parameter to DtsCreator#writeFile to allow it to accept a postprocessor function, a function that accepts the formatted output from the creator and then returns a string with whatever postprocessing applied. It defaults to the identity function, making it optional. A test was also added.
Postprocessing is also useful for anyone who wants to add a banner comment to the top of their generated files, eg. // This file was generated automatically. Do not edit by hand!

@Quramy Done!

Thank you for PR ! I published your changes as v0.6.3 .


Feature Request: Message size from the inbox view

It would be super nice if (like Mailcatcher) it would display the message size in the inbox view. There are times when I've got several similar emails, but the message size alone would make life very nice.

:+1: not sure when i'll have time, but PR's welcome!


Service Interceptors

Informations

Type
Status
Migration

Feature
In Progress
No

Description
Interceptor for service methods that can access other components from tsED
Usage example
Example to use your feature and to improve the documentation after merging your PR
PermissionInterceptor.ts
import { Interceptor, IInterceptor, IInterceptorContext } from "ts-express-decorators";

@Interceptor()
export class PermissionInterceptor implements IInterceptor {
 aroundInvoke(ctx: IInterceptorContext) {
    if (...) {
      return ctx.proceed();
    } else {
      throw new Error('Cannot execute method');
    }
 }
}

UserService.ts
import { Service, Intercept } from 'ts-express-decorators';
import { PermissionInterceptor } from '../interceptors';

@Service()
export class UserService {
   @Intercept(PermissionInterceptor)
   add(userModel) {
     ...
   }
}

Note: Injection of services into classes is not working currently @Romakita  I get the issue that we already discussed. I have this interceptor:

And this is what I get...

Please check the code and suggest changes. Thanks!
Todos

[ ] Fix injection of services in interceptor class
[ ] Tests
[ ] Coverage
[ ] Example
[ ] Documentation

Hi. For me, the interceptor works. Here the branch: https://github.com/Romakita/ts-express-decorators/tree/feat_interceptors
I changed nothing, except the service injected to the interceptor. Are you sure, your documentService is correctly declared as a service ?

@Romakita hmm, maybe I am not building it correctly? I run "npm run build", but in the index.ts I had to modify the imports from @tsed/core and @tsed/common to ./core and ./common? Do you think this might be the problem?
I don't think the services are a prob. because I can inject other services in them, and any other component in my app except the interceptors.

HMm, I could inject the InjectorService but when I try my services it breaks, you might be correct that my services are not ok, but I don't know what is the prob.. I don't quite understand the whole bootup process, maybe something is not correctly wired before the .load?

Maybe is a circular dependency because I am trying to use a service in the interceptor, but I use the interceptor on the services as well..

For me, all of your export is good.
Run a npm install on the Ts.ED directory.
Then, on your project, try to run theses command:
npm link ../path/to/ts-express-decorators/src/core
npm link ../path/to/ts-express-decorators/src/common

Normally @tsed/common and @tsed/core will point to your local ts-express-decorators.

hum... I'll try with a service T.T

Hmm, I cannot find registerService.. could you point me to the right direction?

Strange... when I remove the @Intercept(PermissionInterceptor) from the method in the service I can see the constructor services injected.. it only happens when I am using the interceptor on the methods that I cannot get the services injected...

registerService is on the latest version of Ts.ED. You need to do that:
git remote add upstream git://github.com/Romakita/ts-express-decorators.git
git stash
git rebase upstream/production
git stash pop

Can I have an access to your project ? Because I can't reproduce your bug actually ^^.

I saw your example, try to create another service and inject it in the PermissionInterceptor. For e.g. create
@Service()
export class MyService {
  sayHello() {
   return 'hello';
  }
}

Then instead of
     constructor(parseService: ParseService) {
        console.log(parseService);
     }

do
     constructor(private myService: MyService) {
        console.log(parseService);
     }

I will try to create separate branch on my proj and allow you to check it out. I will have to make the db mock so that you can run it without deps. Btw, I saw that if I comment the usage of the @Intercept the injection works. Seems that whenever I import the PermissionInterceptor and use it inside the @Intercept I get this issue.

@mitevs Here my public key:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDeGYYt9Heeb1Z9R4M1UDVI22rzQzYH9mnOnSQyKyvkMrNNVWOZc8iG+qLHX0ZmRz/BFxSwhWd/0r95mlRjeNr0xBZUYU3u2XgQ7QKmxTTy+1qQf9mVt6y5HMkluqg0Qi6sDVxpT0NhaJVpd3VW//whq4jFBlsNmfc+XK4tscKjT7kHyLTyYM0pqGGFytLgtfdcXA3GZ6R/+b4krsmGbQeGe03y+/1ox8x5ta5egTMyvLeWRaGZqRNFaPyp4Jso2bX0r37E3rTi4OSdOgqc5ro4tNkOBFVRs5i4EClVbDAw5yO673i9yIWw8le73xJhlCTwF+Z3+5E9K+IGBF9GZB5

@Romakita  you can check it out now, here's the repo git@bitbucket.org:code-bros/dip-cv-rest.git

@Romakita you will see the way I import the interceptor, but I had to export them in the interceptors/index.ts then do "export * from './interceptors'" in the services/index.ts and then I need to import the interceptor class from the services/index.ts in order for it to work. Any other way is not working, and I am assuming the issue is circular deps..

Yes, you have circular dependencies due to inheritance between your PermissionService and BaseService. In this case, you need to refactoring your code to avoid this circular dependency.

I've managed to fix it. Let me know if I should proceed with this branch for the feature. Currently I defined the interceptor decorators in my project until this goes in tsED

This functionality is interesting :) PR are welcome.
I've updated the feat_interceptor branch on the Ts.ED project with your last modification.

ok will discard this PR and continue with that branch, will submit PR from it


Problem with torch

I tried to download it like in manual but got this:

So it seems this old version is unavaible now, can't install it from the site. I installed latest version instead (1.11.0) but got this line: ...\Python310\lib\site-packages\torch\nn\parameter.py", line 2, in
from torch._C import _disabled_torch_function_impl
ModuleNotFoundError: No module named 'torch._C'
I don't know what to do with this.

Ok, it seems I needed older Python version. I should've expected that.


Add test for issue #963

I created this test to help to fix the bug.

@rubenrua I've cherry-picked your test into https://github.com/doctrine/mongodb-odm/pull/981 and it passes :)

@malazm +1


[FR] Highlighting components within templates

Would it be possible to highlight components in templates like in Svelte/React?
Vue (Volar)

Svelte

VSCode has default config "editor.semanticHighlighting.enabled": "configuredByTheme" so it may disable by some theme extensions, you can config "editor.semanticHighlighting.enabled": true to force enable it.

Sorry, I didn't find the right Issuse, I used the unplugin-vue-components in the project he can automatically introduced, but this will cause the component color to be red, it seems that volar does not resolve his introduction.

I have to introduce components through import, and he will display the correct color.

@RennCheung make sure you config correctly, see https://github.com/antfu/unplugin-vue-components#typescript

Sorry, it was my negligence.👍

@johnsoncodehk Setting editor.semanticHighlighting.enabled to true fixed this, but I'm curious why is it that the Svelte extension, for instance, shows the right color (green) on components even when editor.semanticHighlighting.enabled is false or "configuredByTheme"?!
Whatever they're doing, can't Volar adopt the same technique to avoid confusion?

but I'm curious why is it that the Svelte extension, for instance, shows the right color (green) on components even when editor.semanticHighlighting.enabled is false or "configuredByTheme"?!

I don't think you can say svelte show correct green color, it just always show green color, but append an error.

On the contrary, I think at least for now, show an error is too radical for vue users, so it just shows a different color but no error.
editor.semanticHighlighting.enabled behavior is designed by vscode, so you can consider blaming vscode instead of volar.

@johnsoncodehk But the point here is that sometimes in Volar the component is properly imported, but it's still red, like:
This is solved by setting editor.semanticHighlighting.enabled to true, but Svelte extension doesn't have this problem even when editor.semanticHighlighting.enabled is false:

@AradAral I did not misunderstand what you mean.
What I mean is that vscode defaults editor.semanticHighlighting.enabled to configuredByTheme  to cause this problem, which is not controlled by Volar.
There is no problem with svelte, just because svelte does not support displaying different colors, not because svelte solves this problem.

What I mean is that vscode defaults editor.semanticHighlighting.enabled to configuredByTheme to cause this problem, which is not controlled by Volar.

Sure :) I'm asking why doesn't Volar color highlighting work as expected regardless of the value of editor.semanticHighlighting.enabled? Like Svelte does?

@AradAral Sorry but I don't really understand your question, do you mean always shows green like svelte? This is probably not in line with the volar design.
In any case, if you think the design of the function is unreasonable, please open an issue to make a reasonable suggestion instead of complaining.

I just found that vscode allow extension setup default config (https://code.visualstudio.com/api/references/contribution-points#contributes.configurationDefaults), this problem should be fixed in next version. Sorry for the confusion!


Add configuration parameters to UI extensions

Add configuration to UI extensions

@mariobodemann Thanks for your comments. Could you please take a look?

By removing the SingletonMap class, the deserializer is not needed so I removed.
I fixed most style issues and the linter warnings


About bibliographical references

Dear @erc-dharma/nusantara-epigraphy,
It seems you haven't really started working on those, but for some of you more advanced in their work, please take the following remarks into account:

the prefix bib: have been forgotten several times for both @source and @target, also it is necessary for the second reference, e.g not bib:Bosch1928_01 Sarkar1971-1972_01 but bib:Bosch1928_01 bib:Sarkar1971-1972_01
some shortTitles are not well formed (2 authors without the +, missing ending number.... ). Be careful with this, any mistake will make impossible retrieving data from Zotero.
@unit="annex haven't been planed by the EG, could @unit="appendix" be enough ?
<citedRange> always take the attribute @unit.

Best,

Thanks a lot.

I am the source for the erroneous bib:Bosch1928_01 Sarkar1971-1972_01, and have now corrected that mistake in the relevant file.
please indicate which files are concerned by the problem of ill-formed shortTitles
yes, @unit="appendix" is fine: which files are concerned?
which files lack @unit on <citedRange>

For @unit:

@aditiagunawan

Batutulis
Huludayeuh
kawali_1a
kebantenan_1
kebantenan_2
Linggawangi.xml

@arlogriffiths

Wurare

@ekobastiawan

Gadhakuti
Maribong
PakisWetan

@m-schoettel

Canggu
Kusmala
Paguhan

@unit="appendix" is only in Canggu

Sorry, Axelle, EG p. 116 says:

by default, <citedRange> will be understood to refer to page numbers, so references to pages can simply be added as the content of this element
use a hyphen to record a range of pages, e.g. <citedRange>12-21</citedRange>
the number of the last page should always be recorded in full, e.g. <citedRange>123-124</citedRange> (not 123-4 or 123-24)
use a comma (followed by a space) to list non-adjacent pages, e.g. <citedRange>12, 24</citedRange>
to refer to an entity other than a page, add the attribute @unit to <citedRange> with one of the following values

So I don't think at least the cases in Wurare.xml are encoding errors.
Or do you think the EG shouild be changed on this point?

For me, it is a mistake. But of course, you can choose to encode as Daniel expect. Your decision.

@ajaniak : can we close this issue now?


OAuth2 authentication method

@bmassemin Just checking-in, are you still working in this PR or is it ready for review?

Not for now, I'll probably resume later


.travis.yml attempt to fix travis error

Travis gets further along with coveralls removed and to my knowledge coveralls was never completely integrated anyway, so I'm merging this.


Is there any possibility of displaying all the test cases with names in Github actions summary

Hi
Hope all is well. Is there any possibility that we can see the test results in a github actions summary like below example.

Thank you !!

That should be doable, but this feature would have to be disabled by default as this can grow very large for many users.
Contribution welcome, should be straight forward to add.


LightGBMClassifier failed with ConnectionRefused spark in spark-submit with cluster mode

Describe the bug
My application based on mmlspark is failed that is launched with spark cluster mode.
Training small dataset is success. However, it is failure to train large dataset.
Spark-submit with client has not any problem. This error, ConnectionRefused, is occurred in cluster mode.
To Reproduce

An application that is train large dataset with LightGBMClassifier is launched with spark cluster mode

Expected behavior
Training is successful.
However. ConnectionRefused Exception is occurred.
Info (please complete the following information):

MMLSpark Version: v0.17
Spark Version: 2.3.0
Spark Platform: EMR

** Stacktrace**
Please post the stacktrace here if applicable

User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 4.0 failed 4 times, most recent failure: Lost task 0.3 in stage 4.0 (TID 256, hdp-d08.my.cluster.com, executor 5): java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.(Socket.java:434)
at java.net.Socket.(Socket.java:211)
at com.microsoft.ml.spark.TrainUtils$.getNetworkInitNodes(TrainUtils.scala:277)
at com.microsoft.ml.spark.TrainUtils$$anonfun$7.apply(TrainUtils.scala:340)
at com.microsoft.ml.spark.TrainUtils$$anonfun$7.apply(TrainUtils.scala:335)
at com.microsoft.ml.spark.StreamUtilities$.using(StreamUtilities.scala:29)
at com.microsoft.ml.spark.TrainUtils$.trainLightGBM(TrainUtils.scala:334)
at com.microsoft.ml.spark.LightGBMBase$$anonfun$4.apply(LightGBMBase.scala:49)
at com.microsoft.ml.spark.LightGBMBase$$anonfun$4.apply(LightGBMBase.scala:49)
at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$5.apply(objects.scala:188)
at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$5.apply(objects.scala:185)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1599)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1587)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1586)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1586)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:831)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1820)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1769)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1758)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2034)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2131)
at org.apache.spark.rdd.RDD$$anonfun$reduce$1.apply(RDD.scala:1029)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
at org.apache.spark.rdd.RDD.reduce(RDD.scala:1011)
at org.apache.spark.sql.Dataset.reduce(Dataset.scala:1610)
at com.microsoft.ml.spark.LightGBMBase$class.train(LightGBMBase.scala:51)
at com.microsoft.ml.spark.LightGBMClassifier.train(LightGBMClassifier.scala:23)
at com.microsoft.ml.spark.LightGBMClassifier.train(LightGBMClassifier.scala:23)
at org.apache.spark.ml.Predictor.fit(Predictor.scala:118)
at com.my.myapp.infrastructure.MyAppTrainer.train(MyAppTrainer.scala:--)
at com.my.myapp.application.MyAppTrainer.train(MyAppTrainer.scala:--)
at com.my.myapp.myApp$.main(MyAppTrainerApp.scala:--)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.my.myapp.util.spark.SparkApp$$anonfun$main$2.apply(SparkApp.scala:--)
at com.my.myapp.util.spark.SparkApp$$anonfun$main$2.apply(SparkApp.scala:--)
at scala.Option.foreach(Option.scala:257)
at com.my.myapp.util.spark.SparkApp$class.main(SparkApp.scala:--)
at com.my.myapp.infrastructure.MyApp$.main(MyApp.scala:--)
at com.my.myapp.MyApp.main(MyApp.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.(Socket.java:434)
at java.net.Socket.(Socket.java:211)
at com.microsoft.ml.spark.TrainUtils$.getNetworkInitNodes(TrainUtils.scala:277)
at com.microsoft.ml.spark.TrainUtils$$anonfun$7.apply(TrainUtils.scala:340)
at com.microsoft.ml.spark.TrainUtils$$anonfun$7.apply(TrainUtils.scala:335)
at com.microsoft.ml.spark.StreamUtilities$.using(StreamUtilities.scala:29)
at com.microsoft.ml.spark.TrainUtils$.trainLightGBM(TrainUtils.scala:334)
at com.microsoft.ml.spark.LightGBMBase$$anonfun$4.apply(LightGBMBase.scala:49)
at com.microsoft.ml.spark.LightGBMBase$$anonfun$4.apply(LightGBMBase.scala:49)
at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$5.apply(objects.scala:188)
at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$5.apply(objects.scala:185)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
If the bug pertains to a specific feature please tag the appropriate CODEOWNER for better visibility
Additional context
I've tried to increase timeout value. However, in cluster mode, the error is reproduced.

@ocworld the connection refused is usually a red herring and it means that one of the workers failed or succeeded early.  How many rows/cols does your dataset have?  Is it unbalanced data and what is the class distribution?  How many nodes do you have in your cluster and what are the specs?

This connection refused error is occurred because of lack of resources on EMR not succeeded early. The number of features are over 300. I tested LightGBM library on python directly on a local server. The amount of memory is not needed to large as mmlspark. I Have a question about why mmlspark consume large memory than using LightGBM library directly?
I try to do repartition(coalesce) on my private spark cluster, it is not need to large memory as it. It is difficult to understand what operations are need to more memory.
Why more memory is needed to use LightGBM on mmlspark?

@ocworld sorry about the error you are encountering.  Would you be able to share the dataset schema and how many rows the dataset has (I understand it has 300 columns).  Also please share the number of nodes (VMs) and total memory on the cluster, as well as the number of executors.  Just to confirm, you are not using dynamic allocation, is that correct?

same problem with mmlspark 1.0.0-rc1 and spark 2.4.0-cdh:
19/10/28 11:31:40 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 4683 19/10/28 11:31:40 INFO executor.Executor: Running task 20.1 in stage 56.0 (TID 4683) 19/10/28 11:31:40 INFO storage.BlockManager: Found block rdd_107_74 remotely 19/10/28 11:31:40 INFO lightgbm.LightGBMClassifier: Successfully bound to port 12424 19/10/28 11:31:40 INFO lightgbm.LightGBMClassifier: LightGBM worker connecting to host: <IP_ADDRESS> and port: 39301 19/10/28 11:31:40 ERROR executor.Executor: Exception in task 20.1 in stage 56.0 (TID 4683) java.net.ConnectException: Connection refused (Connection refused) 	at java.net.PlainSocketImpl.socketConnect(Native Method) 	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) 	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) 	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) 	at java.net.Socket.connect(Socket.java:589) 	at java.net.Socket.connect(Socket.java:538) 	at java.net.Socket.<init>(Socket.java:434) 	at java.net.Socket.<init>(Socket.java:211) 	at com.microsoft.ml.spark.lightgbm.TrainUtils$.getNetworkInitNodes(TrainUtils.scala:299) 	at com.microsoft.ml.spark.lightgbm.TrainUtils$$anonfun$12.apply(TrainUtils.scala:372) 	at com.microsoft.ml.spark.lightgbm.TrainUtils$$anonfun$12.apply(TrainUtils.scala:367) 	at com.microsoft.ml.spark.core.env.StreamUtilities$.using(StreamUtilities.scala:28) 	at com.microsoft.ml.spark.lightgbm.TrainUtils$.trainLightGBM(TrainUtils.scala:366) 	at com.microsoft.ml.spark.lightgbm.LightGBMBase$$anonfun$6.apply(LightGBMBase.scala:145) 	at com.microsoft.ml.spark.lightgbm.LightGBMBase$$anonfun$6.apply(LightGBMBase.scala:145) 	at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$5.apply(objects.scala:188) 	at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$5.apply(objects.scala:185) 	at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836) 	at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836) 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) 	at org.apache.spark.scheduler.Task.run(Task.scala:121) 	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$11.apply(Executor.scala:407) 	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:413) 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 	at java.lang.Thread.run(Thread.java:748) 19/10/28 11:31:40 INFO executor.CoarseGrainedExecutorBackend: Driver commanded a shutdown 19/10/28 11:31:40 INFO memory.MemoryStore: MemoryStore cleared 19/10/28 11:31:40 INFO storage.BlockManager: BlockManager stopped 19/10/28 11:31:41 INFO util.ShutdownHookManager: Shutdown hook called

@yinshurman @imatiach-msft I've tried to change default GC to G1GC. After that, less memory is needed to train a model.
A document about AWS EMR is helpful to me.
https://aws.amazon.com/ko/blogs/big-data/best-practices-for-successfully-managing-memory-for-apache-spark-applications-on-amazon-emr/
@imatiach-msft The number of rows and columns are very various. The count of rows is over a hundred million.

I assume that generateSparseDataset needs very huge memory. I've traced where is a memory peak point with additional logs I appended.
Additionally, is there more methods for less required memory?

below code in LightGBMBooster::translate is very expensive.
val rows = inputRows.toArray

As estimating memory by using java.lang.Runtime, more memory is used after the line than before.

@ocworld interesting find, I need to convert the spark dataframe to a lightgbm dataset in memory.  I wonder if there might be a more efficient way to do the conversion?  Also, what version of mmlspark are you running - I had a bugfix where I only selected the relevant columns (label, features) to reduce memory usage, otherwise all columns were being generated in memory that existed on the dataframe, including ones that were not relevant.

"I assume that generateSparseDataset needs very huge memory. I've traced where is a memory peak point with additional logs I appended."
it just generates the native lightgbm in-memory representation for sparse format.  Was the original dataset sparse or dense?  One problem might be that spark can have mixed sparse/dense row vectors, but lightgbm only has a sparse or dense representation, so I take the first row and depending on whether it is sparse or dense convert to the lightgbm representation.  There is definitely a more efficient way to do this, since in the worst case scenario the first row can be sparse and the rest of the rows dense, but there is a tradeoff because then you need to use more compute time to figure out what is the most efficient in-memory representation.

@imatiach-msft Thank you for your response.
Firstly, 1.0.0-rc1 is more efficient as aspect of memory I tested. I used 0.17 and 0.18.1. Now, the package is updated to 1.0.0-rc1. I've tested training on the new version using a dataset that is also tested on old versions. In the same cluster specs and spark parameters as versions raising exception, the new version, 1.0.0-rc1, do not occur error. I don't know exactly different between these versions.
Secondly, I think that a big memory consumer is not generateSparseDataset but inputRows.toArray in LightGBMBooster::translate. When first time, I've checked only committed logs. After that, I've appended more logs for estimating used memory. For calculating it, java.lang.Runtime is imported to use total memory and free memory. In my dataset, after collecting about 6~7GB parquet files on a worker as array, memory usage is increasing over 40GB. This memory usage is already high before generateSparseDataset.
When I tested LightGBM on python, I use a dataset as binary file or svm format. I think, so, it is more efficient than mmlspark as aspect of memory. Is it possible for LightGBM to support stream data or convert rdd to one or more binary format data structure?

In addition, when assign executor memory, it should be considered that is spark.memory.fraction.

@ocworld great questions.  This was the main PR that reduced memory usage which only selected the relevant columns needed:
https://github.com/Azure/mmlspark/pull/586
Version 0.17 did not contain this fix, it was in 0.18.  Without this fix any temporary columns used to create the features column (or other columns not used by lightgbm) were being generated and kept in memory which for some users was causing OOM errors.
LightGBM requires all of the data to be in-memory during training, it does not support a streaming API.  Hence, we need to store the original data in spark and also translate to native dataset representation that LightGBM can train on.  Before translating to native, I do inputRows.toArray, which increases memory usage even more (it seems that inputRows copies the actual spark dataframe).  I expect there to be roughly 3X memory usage of the original data.  However, I think we might be able to release rows after inputRows.toArray as we build up the native dataset.  I don't see how I could build up the native data with inputRows streamed (and not call .toArray) unless I get the length of the inputRows ahead of time and pre-allocate the native arrays, but calling size on the iterator goes through the iterator and I would need to re-iterate over the inputRows again (which I believe I might only be able to do with another mapPartitions call?).

@imatiach-msft I still face this issue in version 0.18.1 . Is there any particular fix or solutionrelated to this ?


Update game-scene.ts

update candy crush

@quan-nguyen-thuc06 Thank you for your PR. You touch multiple games, would you mind to make smaller PR for each game?


Compile error on Mac14.6: 'numpy/arrayobject.h' file not found

Environment: MacOS 14.6, M3 Pro chip
Darwin menghuas-MacBook-Pro.local 23.6.0 Darwin Kernel Version 23.6.0: Fri Jul  5 17:55:37 PDT 2024; root:xnu-10063.141.1~2/RELEASE_ARM64_T6030 arm64
Get a compile error when running './getdeps.sh'
In file included from libs/python/src/numpy/ufunc.cpp:8:
./boost/python/numpy/internal.hpp:29:10: fatal error: 'numpy/arrayobject.h' file not found
29 | #include <numpy/arrayobject.h>
|          ^~~~~~~~~~~~~~~~~~~~~
This is the full output log file:
output.log
Thanks a lot!

I solved this problem by adding the numpy/core/include file into C_INCLUDE_PATH and CPLUS_INCLUDE_PATH, but met another compile error when building the libevent.
FAILED: lib/libevent_openssl-2.1.7.dylib
Undefined symbols for architecture x86_64:
"_BIO_clear_flags", referenced from:
_bio_bufferevent_write in bufferevent_openssl.c.o
_bio_bufferevent_read in bufferevent_openssl.c.o
ld: symbol(s) not found for architecture x86_64
cc: error: linker command failed with exit code 1 (use -v to see invocation)
This is the full output log file:
libevent_error.log
Thanks a lot!


[20.03] azure-cli: 2.0.81 -> 2.1.0

Motivation for this change
backporting as it may contain security fixes
backport of #80678
(cherry picked from commit 4dd173560d472ff15703d6b5225f08f6dfeb771a)
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

[1 built, 0.0 MiB DL]
https://github.com/NixOS/nixpkgs/pull/80687
1 package built:
azure-cli

aarch failure is not related to bump


remove libjpeg of SFML CONTROL file

according to https://en.sfml-dev.org/forums/index.php?topic=24009.msg163052#msg163052
libjpeg is no longer a dependancies of sfml :
"libjpeg is no longer a dependency of SFML, but since you use the old FindSFML.cmake it still thinks it is."

Thanks!


Scrolling through a list of draggable elements

Making a 'snappie' style version of draggable here: http://codepen.io/zephyr/pen/oXpOdW
Running into a couple of bugs however.

On mobile devices, even though the drag is constrained to the x axis, it is tough to get vertical scroll to work for it (even though the snap works perfectly, except it seems to be a bit jagged on mobile, though that may just be my amazon fire phone). I can't think of a way to make it detect if it is scrolling or not (vertically only). tl;dr scroll in vertical, drag in horizontal.

On desktop versions, if the mouse escapes the screen while the element is being dragged, the user needs to click back into the window to stop it from dragging. This also occurs if the user drags outside of the mobile screen, and then needs to drag again to fire the dragEnd event (might be just on amazon fire phone)

Great library though!

Draggabilly  does not account for vertical mobile scrolling — like Flickity does. If you have a page full of draggable elements, it will be tough to scroll.
This occurs because of the CodePen is in an iframe. Try the debug version

Ah ok then, thanks for the help!

I have a similar problem to 1). For me a good solution would be to provide an option to only activate dragging after long press. If you don't have the time to implement this feature, I would try. Please let me know.

Having tried Flickity, it fits perfectly for what I wanted to do (thanks desandro, ended up buying a licence as well), have you checked that out?

I'd like to implement drag&drop between multiple scrollable containers. Unfortunately Flickity does not help here. :confused:

any news on this?
codepen links are broken
kabalage, did you achieve to only activate dragging after long press? do you have an example?
i have a page full of draggable elements and new Draggabilly(...) kills the vertical scroll on mobile devices, even with the draggie element .disable()
thanks desandro for your libraries.


encode() is O(n^2)

    for &val in input.clone() {
        let mut carry = val as usize;
        for byte in &mut output[..] {
            carry += (*byte as usize) << 8;
            *byte = (carry % 58) as u8;
            carry /= 58;
        }
        while carry > 0 {
            output.push((carry % 58) as u8);
            carry /= 58;
        }
}

This is insanely slow on large inputs

I would highly recommend not using base58 for large inputs in any case, because the encoding maps each decoded byte to an irrational number of encoded bits it's impossible to stream the data in chunks, you must encode/decode the entire input before you can get the first byte. If you're transferring a large amount of data the upsides of base58 vs base64 don't really apply, and the latter has an exact mapping from 3 decoded bytes to 4 encoded bytes that allows streaming/chunked encoding/decoding.


port Pregel segmented buffers

Scope & Purpose
(Can you describe what functional change your PR is trying to effect?)

[x] Bug-Fix for devel-branch (i.e. no need for backports?
[x] The behavior in this PR can be (and was) manually tested (support / qa / customers can test it)
[x] The behaviour change can only be verified via automatic tests

Testing & Verification
This PR adds tests that were used to verify all changes:

[x] Added Regression Tests (Only for bug-fixes)
[x] Added new C++ Unit Tests (Either GoogleTest or Catch-Test)
[x] Added new integration tests (i.e. in shell_server / shell_server_aql)

Additionally:

[x] I ensured this code runs with ASan / TSan or other static verification tools

http://jenkins01.arangodb.biz:8080/view/PR/job/arangodb-matrix-pr-linux/6183/
Documentation

[x] Added a Changelog Entry
[x] Added a new section in the Manual

http://jenkins01.arangodb.biz:8080/view/PR/job/arangodb-matrix-pr-linux/6206/


add more detailed documentation

[ ] operators
[ ] stages
[ ] preprocess / forward / postprocess
[ ] this
[ ] serializers

also make the examples in doc work.
pytest --doctest-modules  fails at the moment.

if needed also make the examples in doc work. pytest --doctest-modules fails at the moment.

I've started going through the doctest examples to see if I can get them to pass in branch https://github.com/lf1-io/padl/tree/doc_tests

Examples for the frontpage website:

Side-by-side preprocess/ dataloader old way vs. padl
loading model old way and new way
...

building model old way
class SubNet1(torch.nn.Module):
    def forward(self, x):
          ...

class SubNet2(torch.nn.Module):
    def forward(self, x):
          ...

class SuperNet(torch.nn.Module):
    def __init__(self):
            self.subnet1 = SubNet1()
            self.subnet2 = SubNet2()
    def forward(self, x):
          x = self.subnet1(x)
          x = self.subnet2(x)
          return x

new way:
@transform
class SubNet1(torch.nn.Module):
    def forward(self, x):
          ...

@transform
class SubNet2(torch.nn.Module):
    def forward(self, x):
          ...

SuperNet = (
     SubNet1
     >> SubNet2
)

Hmm I think that's cheating a bit, you could just do
SuperNet = torch.nn.Sequential(
    SubNet1(),
    SubNet2()
)


UDP scans with bandwidth (-B) option incorrectly compute probe size and send at incorrect rate

(Historically related to #488)
I've noticed several issues with UDP scans that attempt to constrain the amount of bandwidth used during a scan, and it all seems to do with an improper computation of the probe size which is necessary to compute the proper packet rate to stay within the specified bandwidth option (-B).
For any UDP study that specifies a non-empty probe (--probe-args:text:<text>, --probe-args:file:<some file>, or --probe-args:template:<some template>), the size of the probe is always calculated as 84 bytes and ignores the UDP payload.
For example:
zmap -p 5060 -M udp -O csv -f timestamp_ts,saddr,sport,daddr,dport,ipid,ttl,data -v5 --probe-args=text:ffffffffff -B 1M
zmap -p 5060 -M udp -O csv -f timestamp_ts,saddr,sport,daddr,dport,ipid,ttl,data -v5 --probe-args=file:/any/file -B 1M
zmap -p 5060 -M udp -O csv -f timestamp_ts,saddr,sport,daddr,dport,ipid,ttl,data -v5 --probe-args=template:/any/file -B 1M

All result in using a probe size of 84:
Feb 05 10:29:16.895 [DEBUG] send: using bandwidth 1000000 bits/s for 84 byte probe, rate set to 1488 pkt/s

For the file and text option, it should compute the correct size of the overall probe by including the size of the text/file.  For templates it is a little harder because the template options used in the template are what ultimately dictate the size of the probe, and these are computed on a per-target basis.  In this case it may make sense to compute the rate based on the maximum size of the evaluated template by tracking the explicit size of the template options when possible, and when a field is src/target IP/port/etc dependent, pick a sane upper bound for each field.

@jhart-r7 can you confirm that this is fixed in master now that #499 is merged?

Sure -- I think you mean #506

whoops! Yep, #506.

For text probe types, it still looks to incorrectly set the size to 84:
$  ./src/zmap -n 1 -p 5060 -M udp -O csv -f timestamp_ts,saddr,sport,daddr,dport,ipid,ttl,data -v5 --probe-args=text:ffff -B 1M -G 00:00:00:00:00:00 2>&1 |grep "using band"

Mar 28 12:30:50.697 [DEBUG] send: using bandwidth 1000000 bits/s for 84 byte probe, rate set to 1488 pkt/s
$  ./src/zmap -n 1 -p 5060 -M udp -O csv -f timestamp_ts,saddr,sport,daddr,dport,ipid,ttl,data -v5 --probe-args=text:ffffffff -B 1M -G 00:00:00:00:00:00 2>&1 |grep "using band"

Mar 28 12:30:55.677 [DEBUG] send: using bandwidth 1000000 bits/s for 84 byte probe, rate set to 1488 pkt/s

For file probe types, it now looks correct in that the size of the probe increases directly inline with the file:
$  echo -n thisprobeis22byteslong > probe.txt
$  ./src/zmap -n 1 -p 5060 -M udp -O csv -f timestamp_ts,saddr,sport,daddr,dport,ipid,ttl,data -v5 --probe-args=file:probe.txt -B 1M -G 00:00:00:00:00:00 2>&1 |grep "using band"
Mar 28 12:34:51.416 [DEBUG] send: using bandwidth 1000000 bits/s for 88 byte probe, rate set to 1420 pkt/s
$  echo -n thisprobeis32byteslong1234567890 > probe.txt
$  ./src/zmap -n 1 -p 5060 -M udp -O csv -f timestamp_ts,saddr,sport,daddr,dport,ipid,ttl,data -v5 --probe-args=file:probe.txt -B 1M -G 00:00:00:00:00:00 2>&1 |grep "using band"
Mar 28 12:35:27.844 [DEBUG] send: using bandwidth 1000000 bits/s for 98 byte probe, rate set to 1275 pkt/s

For templates, the size of the probe seems to scale relative to the non-template options used in the template:
$  echo -en '${RAND_ALPHA=16}\nFoo:bar' > test.tpl
[ jhart@LAX-MBP-1576 (03/28/18 12:54:29) ~/rapid7/zmap  ]
$  ./src/zmap -n 1 -p 5060 -M udp -O csv -f timestamp_ts,saddr,sport,daddr,dport,ipid,ttl,data -v5 --probe-args=template:test.tpl -B 1M -G 00:00:00:00:00:00 2>&1 |grep "using band"
Mar 28 12:54:34.688 [DEBUG] send: using bandwidth 1000000 bits/s for 90 byte probe, rate set to 1388 pkt/s
$  echo -en '${RAND_ALPHA=32}\nFoo:bar' > test.tpl
$  ./src/zmap -n 1 -p 5060 -M udp -O csv -f timestamp_ts,saddr,sport,daddr,dport,ipid,ttl,data -v5 --probe-args=template:test.tpl -B 1M -G 00:00:00:00:00:00 2>&1 |grep "using band"
Mar 28 12:54:40.504 [DEBUG] send: using bandwidth 1000000 bits/s for 90 byte probe, rate set to 1388 pkt/s
$  echo -en '${RAND_ALPHA=32}\nFoo:barbaf' > test.tpl
$  ./src/zmap -n 1 -p 5060 -M udp -O csv -f timestamp_ts,saddr,sport,daddr,dport,ipid,ttl,data -v5 --probe-args=template:test.tpl -B 1M -G 00:00:00:00:00:00 2>&1 |grep "using band"
Mar 28 12:54:47.906 [DEBUG] send: using bandwidth 1000000 bits/s for 93 byte probe, rate set to 1344 pkt/s

Re the first one, is this actually incorrect given that the minimum frame size is 84 bytes? If both of those sizes are < 84 bytes, it would be correct to mark them as 84 bytes, no?

Oh right, correct.  I had that in my head when I tested file and template, but forgot to double check that for text.  It now seems to grow relative to the size of the provided text:
$   ./src/zmap -n 1 -p 5060 -M udp -O csv -f timestamp_ts,saddr,sport,daddr,dport,ipid,ttl,data -v5 --probe-args=text:fffffffffffffffffffff -B 1M -G 00:00:00:00:00:00 2>&1 |grep "using band"

Mar 28 13:07:58.393 [DEBUG] send: using bandwidth 1000000 bits/s for 87 byte probe, rate set to 1436 pkt/s
$   ./src/zmap -n 1 -p 5060 -M udp -O csv -f timestamp_ts,saddr,sport,daddr,dport,ipid,ttl,data -v5 --probe-args=text:ffffffffffffffffffffffffffffffffffffff -B 1M -G 00:00:00:00:00:00 2>&1 |grep "using band"

Mar 28 13:09:25.647 [DEBUG] send: using bandwidth 1000000 bits/s for 104 byte probe, rate set to 1201 pkt/s

Alright, so if I understand right the only issue is that the length of the templated values isn't being calculated correctly? Is that what you're seeing? Are templated values always a static length? Can we just run the templating once with a fake IP in global_init and use that? Or is this more complicated and/or intractable?

@zakird yes, the length of templated values seems to be calculated incorrectly.  It seems as if it is coming from the length of the string that indicates the value.
Find that after 18-bytes we've gone beyond the minimum frame size:
$  echo -n '123456789012345678' > test.tpl
$  ./src/zmap -n 1 -p 5060 -M udp -O csv -f timestamp_ts,saddr,sport,daddr,dport,ipid,ttl,data -v5 --probe-args=template:test.tpl -B 1M -G 00:00:00:00:00:00 2>&1 |grep "using band"
Mar 29 09:32:04.780 [DEBUG] send: using bandwidth 1000000 bits/s for 84 byte probe, rate set to 1488 pkt/s
$  echo -n '1234567890123456789' > test.tpl
$  ./src/zmap -n 1 -p 5060 -M udp -O csv -f timestamp_ts,saddr,sport,daddr,dport,ipid,ttl,data -v5 --probe-args=template:test.tpl -B 1M -G 00:00:00:00:00:00 2>&1 |grep "using band"
Mar 29 09:32:09.563 [DEBUG] send: using bandwidth 1000000 bits/s for 85 byte probe, rate set to 1470 pkt/s

Now add 8 random alpha to the template.  We expect a length of 92 now (84 + 8), but get 99 instead:
$  echo -n '123456789012345678${RAND_ALPHA=8}' > test.tpl
$  ./src/zmap -n 1 -p 5060 -M udp -O csv -f timestamp_ts,saddr,sport,daddr,dport,ipid,ttl,data -v5 --probe-args=template:test.tpl -B 1M -G 00:00:00:00:00:00 2>&1 |grep "using band"
Mar 29 09:32:44.931 [DEBUG] send: using bandwidth 1000000 bits/s for 99 byte probe, rate set to 1262 pkt/s

Adding another 8 random alpha, we see that the length grows 15 instead of 8:

$  echo -n '123456789012345678${RAND_ALPHA=8}${RAND_ALPHA=8}' > test.tpl
$  ./src/zmap -n 1 -p 5060 -M udp -O csv -f timestamp_ts,saddr,sport,daddr,dport,ipid,ttl,data -v5 --probe-args=template:test.tpl -B 1M -G 00:00:00:00:00:00 2>&1 |grep "using band"
Mar 29 09:33:13.001 [DEBUG] send: using bandwidth 1000000 bits/s for 114 byte probe, rate set to 1096 pkt/s

In this case, 15 is the length of `${RANDOM_ALPHA=8}`.

No, the replaced tokens vary according to what is specified:
See that when we go from 8 to 9 random, the length doesn't change:
$  echo -n '123456789012345678${RAND_ALPHA=8}' > test.tpl
$  ./src/zmap -n 1 -p 5060 -M udp -O csv -f timestamp_ts,saddr,sport,daddr,dport,ipid,ttl,data -v5 --probe-args=template:test.tpl -B 1M -G 00:00:00:00:00:00 2>&1 |grep "using band"
Mar 29 10:26:55.209 [DEBUG] send: using bandwidth 1000000 bits/s for 99 byte probe, rate set to 1262 pkt/s
$  echo -n '123456789012345678${RAND_ALPHA=7}' > test.tpl
$  ./src/zmap -n 1 -p 5060 -M udp -O csv -f timestamp_ts,saddr,sport,daddr,dport,ipid,ttl,data -v5 --probe-args=template:test.tpl -B 1M -G 00:00:00:00:00:00 2>&1 |grep "using band"
Mar 29 10:27:03.680 [DEBUG] send: using bandwidth 1000000 bits/s for 99 byte probe, rate set to 1262 pkt/s

See that when we change the length of the token, the size of probe changes:
$  echo -n '123456789012345678${RAND_ALPHA=10}' > test.tpl
$  ./src/zmap -n 1 -p 5060 -M udp -O csv -f timestamp_ts,saddr,sport,daddr,dport,ipid,ttl,data -v5 --probe-args=template:test.tpl -B 1M -G 00:00:00:00:00:00 2>&1 |grep "using band"
Mar 29 10:27:10.623 [DEBUG] send: using bandwidth 1000000 bits/s for 100 byte probe, rate set to 1250 pkt/s
$  echo -n '123456789012345678${RAND_ALPHA=123}' > test.tpl
$  ./src/zmap -n 1 -p 5060 -M udp -O csv -f timestamp_ts,saddr,sport,daddr,dport,ipid,ttl,data -v5 --probe-args=template:test.tpl -B 1M -G 00:00:00:00:00:00 2>&1 |grep "using band"
Mar 29 10:27:21.771 [DEBUG] send: using bandwidth 1000000 bits/s for 101 byte probe, rate set to 1237 pkt/s

As you can see, by adding 2 more characters to the token, the length of the probe increases by 2.

Sure, but it will always be 9 if specified that way? There aren't tokens that are sometimes 8 and sometimes 9 depending on the server they're getting sent to?

I don't know.  I haven't examined everything that is sent out.  I am merely observing the calculated size that is displayed at startup, comparing it to what I expect and seeing how it changes as I use different template tokens.

Templated fields can vary in size depending on the field type and destination. The fixed length field-types are:
SADDR_N: Source IP address in network byte order
DADDR_N: Destination IP address in network byte order
SPORT_N: UDP source port in netowrk byte order
DPORT_N: UDP destination port in network byte order
RAND_BYTE: Random bytes from 0-255
RAND_DIGIT: Random digits from 0-9
RAND_ALPHA: Random mixed-case letters (a-z)
RAND_ALPHANUM: Random mixed-case letters (a-z) and numbers

The variable length fields are
SADDR: Source IP address in dotted-quad format
DADDR: Destination IP address in dotted-quad format
SPORT: UDP source port in ascii format
DPORT: UDP destination port in ascii format

So, aside from whatever bug is screwing with UDP in general, we also can't rely on bandwidth calculations being consistent throughout the lifetime of the program. Alternatively, we could just do bandwidth calculations using the max size of each of those fields (5 for ports, 15 for dotted-quads).

Don't we allocate a buffer size in the beginning at least for how big the UDP packet is going to be? Can we use that to estimate?

Yes, we allocate based on the max size of each field.

This is now fixed as of #669 and 04e6c51


Problem with nested routes

I can't get my route handler called if it is nested.
This calls RobotsDetail at /robots/1 just as I want:
var routes = (
  <Route handler={App} path="/">
    <DefaultRoute name="home" handler={Home}/>
    <Route name="robots-index" path="/robots/" handler={RobotsIndex}/>
    <Route name="robots-detail" path="/robots/:id" handler={RobotsDetail}/>
  </Route>
);

But this somehow calls RobotsIndex at /robots/1 and not RobotsDetail:
var routes = (
  <Route handler={App} path="/">
    <DefaultRoute name="home" handler={Home}/>
    <Route name="robots-index" path="/robots/" handler={RobotsIndex}>
      <Route name="robots-detail" path="/robots/:id" handler={RobotsDetail}/>
      <DefaultRoute handler={RobotsIndex}/>
    </Route>
  </Route>
);

I also tried with relative path. Same result. RobotsDetail is never called.
var routes = (
  <Route handler={App} path="/">
    <DefaultRoute name="home" handler={Home}/>
    <Route name="robots-index" path="/robots/" handler={RobotsIndex}>
      <Route name="robots-detail" path=":id" handler={RobotsDetail}/>
      <DefaultRoute handler={RobotsIndex}/>
    </Route>
  </Route>
);

I also tried this without DefaultRoute directive. No effect to this issue.
I don't understand why. Everything seems correct to me.

I discovered my problem. Parent routes are always called. Thats the intent. But child components need to have repeated <Router.RouteHandler/> to get rendered.

Glad you figured it out, don't forget to read the overview for some basic
concepts like this :)
https://github.com/rackt/react-router/blob/master/docs/guides/overview.md
On Wed, Jan 14, 2015 at 5:55 AM, Ivan Kleshnin<EMAIL_ADDRESS>wrote:

Closed #687 https://github.com/rackt/react-router/issues/687.
—
Reply to this email directly or view it on GitHub
https://github.com/rackt/react-router/issues/687#event-218719805.

@ryanflorence , the link is no longer available.
I have exactly the same question. Is there a way to fix it?

Check the docs https://github.com/rackt/react-router/tree/master/docs
For questions and support, please visit our channel on Reactiflux or Stack Overflow. The issue tracker is exclusively for bug reports and feature requests.


macOS: Allow to specify a name for the TUN interface

Thanks for reviewing, changes look good to me.


JS installation in Rails 4

I successfully installed materializecss in rails with the min.js file and it works great.

However, if I separately load the js files, the javascript parts fail to functioning.

Errors include:
ReferenceError: Can't find variable: Picker
(anonymous function)picker.date.self-5924b278beda2df8570f137f72b37e2d0f516acbc11a0994d1abb74380f078ef.js:18
global codepicker.date.self-5924b278beda2df8570f137f72b37e2d0f516acbc11a0994d1abb74380f078ef.js:20
localhost:170

[Error] TypeError: undefined is not a function (evaluating 'Vel(newToast, { "top" : "0px", opacity: 1 }, {duration: 300,
      easing: 'easeOutCubic',
      queue: false})')
  toast (toasts.self-188603d7c1c8e1177b838234026094fb90a0fe57c94bf0ffa1082dd33619a6cc.js, line 22)
  onclick (localhost, line 167)

Any advice?

This is because there is a concat order. Take a look at our Gruntfile.js.


update

Before opening your pull request, please make sure that you've:

[ ] signed Uber's Contributor License Agreement;
[ ] added tests to cover your changes;
[ ] run the test suite locally (make test); and finally,
[ ] run the linters locally (make lint).

Thanks for your contribution!

update


[API] Move project activity to a new /activity route

Currently when fetching a Project from /projects/[id] we also fetch recent activity for that project.
We have identified this as a major performance cost in #1360 and #1367 - I also anticipate more features around sorting, filtering and exploring that activity.
For these reasons I think it makes sense to treat Activity as it's own resources and serve it from a sub-path
/projects/[id]/activity
This is a breaking change with a new API version. A separate ticket will be needed to consume this new activity feed on the frontend.

@biwano here is a new issue for you when you are free of Klima Data tasks

@Atmosfearful If I understand the release process I will make 3 pull requests:

[X] Modify the API to Create the new activity route: #1888
[ ] Modify the UI to use the new activity route
[ ] Modify the API to remove activity information when querying projects


Add Australian CDR conformance test

Adds a conformance suite testing profile for the Australian CDR.
Note that currently this has to be tested manually; encryption keys for id token encryption need to be provided separately, and supported claims need to be updated in oidf.json.

@shaneboulden Thank you!


tempInterpol1/tempInterpol2

Reported by beutlich on 18 Aug 2014 13:10 UTC
Modelica.Math.tempInterpol1 and Modelica.Math.tempInterpol2 are marked as obsolete but are still used in Mechanics libraries. Calls should be substituted by Modelica.Math.Vectors.interpolate or components of type Modelica.Blocks.Tables.CombiTable1Ds.

Migrated-From: https://trac.modelica.org/Modelica/ticket/1553

Comment by beutlich on 18 Aug 2014 19:05 UTC
Modelica.Math.tempInterpol2 was only called by Modelica.Mechanics.Rotational.Components.LossyGear. This is avoided by 05e9afb65ede0b155fdb46cbc5dfef510a066127 in a backwards-compatible way.

Comment by beutlich on 18 Aug 2014 19:35 UTC
6f3ead3d45db48d15b47372f5e7823744a655d03 removes any calls of Modelica.Math.tempInterpol1 in a backwards-compatible way.

Comment by beutlich on 20 Aug 2014 11:35 UTC
For backward compatibility, CombiTable1Ds must not be used since tempInterpol1/tempInterpol2 allows internal abscissa points to be monotically increasing whereas CombiTable1Ds requires strictly increasing points.

Modified by beutlich on 28 Sep 2015 11:21 UTC

Changelog modified by beutlich on 28 Sep 2015 11:21 UTC
Remove calls to obsolete functions Modelica.Math.tempInterpol1 and Modelica.Math.tempInterpol2

Modified by beutlich on 3 Nov 2015 15:08 UTC

Changelog modified by beutlich on 3 Nov 2015 15:08 UTC
Removed calls to obsolete functions Modelica.Math.tempInterpol1 and Modelica.Math.tempInterpol2


Using BenchmarkCI in Buildkite pipelines

I tried using following script for running benchmarks on buildkite pipline
using Pkg

# Activate `benchmark`
Pkg.activate("benchmark")
Pkg.develop(path=".") # Add parent Project.toml to development env
Pkg.instantiate()
Pkg.build()
Pkg.precompile()

# Start benchmarking
using BenchmarkCI
BenchmarkCI.judge()
BenchmarkCI.postjudge()

But it errors with:
Activating environment at `/<buildkite-path>benchmark/Project.toml`
:
Installed Parsers ──────────────────── v2.1.2
:
✓ Parsers
:
 15 dependencies successfully precompiled in 12 seconds (58 already precompiled)
:
[ Info: Using existing manifest file.
--
  | PkgBenchmark: Running benchmarks...
  | Activating environment at `<buildkite-path>/benchmark/Project.toml`
  | PkgBenchmark: creating benchmark tuning file /<buildkite-path>/benchmark/tune.json..
(1/3) tuning "new"...
:
:
PkgBenchmark: benchmark results written to .benchmarkci/result-target.json

  | PkgBenchmark: Running benchmarks...
  | ERROR: LoadError: LoadError: ArgumentError: Package Parsers [69de0a69-1ddd-5017-9359-2bf0b02dc9f0] is required but does not seem to be installed:
  | - Run `Pkg.instantiate()` to install all recorded dependencies.

( : to be replaced with entire package list)
The issue isn't specific to Parsers, but its just something it is trying to find first.

BenchmarkCI does an impressive job with Github Actions.
If this could be reused with Buildkite runners (which is widely being adopted), it would be great.
While here, postjudge() is something to inspect if it can post even from a buildkite pipeline

Hmm... It's a bit puzzling that the error occurs at such a late phase. Is this from a public repository? Can I see a CI log?

Sadly its a private repo.


upgrade this apk to the latest LW platform

can we please have this apk upgraded to the latest LW platform/version.
currently this apk is not working for any of the LW platform with 28+ android Version.
i have fixed this issue locally and its working for me.
if someone can help to Fix and Maintain this apk for LW platforms that would be great
-sarath

What is an "LW platform"?
If you need to measure Android audio latency then you may want to use OboeTester instead. It is actively maintained.
https://github.com/google/oboe/tree/master/apps/OboeTester/docs

LW platform i was refering to "Wear OS" and "China Version of Wear OS"
sure will give a try on OboeTester apk on Wear OS and China Version of Wear OS


Re-investigate JNDI contributors

Last time I looked into the PVC JNDI system (the com.ibm.pvc.jndi.provider.java bundle), it seemed like there wasn't really a great way to dynamically provide objects on lookup. For example, if I bound the ManagedExecutorService, that'd be cached internally and used for all apps' lookups, which wouldn't go well.
Still, maybe it is indeed possible, and it's at least work another check. The current ad-hoc mechanism of trying to bind and unbind names at all the right times is janky at best.

It would also be possible, at least for the Concurrency elements, to just bind one object at server start, and then have that object delegate to the actual contextual objects in each method. It'd be a little less efficient per-call, but shouldn't be bad.

I found the PVC endpoints to be a bit fiddly. For now, at least, I modified the Concurrency bits to go the delegate route, binding them in the Activator and leaving them there permanently. That should allow for arbitrary code to look them up and use them with less risk of the constant push/pop causing trouble.


[Bug]: WARNING: can't find /ragflow/rag/res/broker.tm

Is there an existing issue for the same bug?

[X] I have checked the existing issues.

Branch name
main
Commit ID
7a36d25
Other environment information
pop_os 22.04
docker 26.0
Intel i7-12800h
32gb

Actual behavior
(base) hitesh@whiskey:~/ragflow/docker$ docker logs -f ragflow-server
[HUQIE]:Build default trie
[HUQIE]:Build trie /ragflow/rag/res/huqie.txt
WARNING:root:Realtime synonym is disabled, since no redis connection.
[WARNING] Load term.freq FAIL!
____                 ______ __

/ __ \ ____ _ ____ _ / // /  _      __
/ // // __ // __ // /   / // __ | | /| / /
/ , // // // // // /  / // // /| |/ |/ /
// || _,/ _, ///    // _/ |/|_/
/____/
ERROR:dashscope:Request: https://dashscope.aliyuncs.com/api/v1/services/aigc/text-generation/generation failed, status: 401, message: Invalid API-key provided.
INFO:werkzeug:WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.

Running on all addresses (<IP_ADDRESS>)
Running on http://<IP_ADDRESS>:9380
Running on http://<IP_ADDRESS>:9380
INFO:werkzeug:Press CTRL+C to quit
WARNING:root:Realtime synonym is disabled, since no redis connection.
[WARNING] Load term.freq FAIL!
WARNING: can't find /ragflow/rag/res/broker.tm
WARNING: can't find /ragflow/rag/res/broker.tm
WARNING: can't find /ragflow/rag/res/broker.tm
WARNING: can't find /ragflow/rag/res/broker.tm
WARNING: can't find /ragflow/rag/res/broker.tm
WARNING: can't find /ragflow/rag/res/broker.tm
WARNING: can't find /ragflow/rag/res/broker.tm
WARNING: can't find /ragflow/rag/res/broker.tm

Expected behavior
____                 ______ __

/ __ \ ____ _ ____ _ / // /  _      __
/ // // __ // __ // /   / // __ | | /| / /
/ , // // // // // /  / // // /| |/ |/ /
// || _,/ _, ///    // _/ |/|_/
/____/

Running on all addresses (<IP_ADDRESS>)
Running on http://<IP_ADDRESS>:9380
Running on http://<IP_ADDRESS>:9380
INFO:werkzeug:Press CTRL+C to quit

Steps to reproduce
I followed the instructions for docker:

$ git clone https://github.com/infiniflow/ragflow.git
$ cd ragflow/docker
$ docker compose up -d

Additional information
No response

Thans for following!

For this issue, please configure your own LLM supplier and API-Key at the 'docker/service_config.yaml' in the following part:

llm_api_key_setup.md has the detail.

WARNING: can't find /ragflow/rag/res/broker.tm
Ignore this warning please. Once you uploaded documents in the system, this will dismiss.

WARNING: can't find /ragflow/rag/res/broker.tm
Ignore this warning please. Once you uploaded documents in the system, this will dismiss.

@KevinHuSh service_config.yaml is not mentioned at all in https://github.com/infiniflow/ragflow/blob/main/docs/quickstart.md. So is this an error, a requirement or a bug? I get this "error" starting ragflow-server and the server doesn't work.


Catch exceptions in Streamlet operators

@nwangtw was there an issue that was fixed by adding the try catches?

No real known issue, but the current operators don't handle failures and I feel the try/catch could be necessary. Or it was by design that Streamlet operators couldn't fail?

@huijunw Do you have any input on this merge request?


Indented same time appointment check. Speech correct in logs but speech blank in api.ai?

what is the issue?
" Speech correct in logs but speech blank in api.ai" <meaning?


Test suite failing due to deprecated --rsyncdir

pytest-xdist 3.0.2 deprecated support for --rsyncdir. As a result, Setuptools test suite now fails early with an INTERNALERROR when that deprecation warning is hit while -W error is indicated (example).

https://github.com/pytest-dev/pytest-xdist/issues/825


Add styling and animation to links and containers.

Add styling and underline animation to links, and change projects to containers with background and shadow.

I didn't see that @bheff-work had left comments on the base branch before I created this CL; sorry about that, I'll address those comments. Part of the reason for creating this CL was also to practice creating a branch that is sourced from another branch, rather than master -- I am looking to understand how to do this before I start working with others on the same project.

I am closing this CL as it contains non-crucial changes to my portfolio pages, and I am now moving on to week 3 in the tutorial sequence.


change default place holder

Hi,
Thanks for this editor and it's really useful. I just wanted to ask how to change defaultPlaceholder or disable AI side of editor?
Thanks again @steven-tey

Set the defaultValue to an empty string, for e.g: <Editor defaultValue="" />.

default

You can see on image it's displaying already the text
I updated like you said.

 @sarimrmalik

Yeap, this is good. This is the placeholder text to get the user started, if you click into the editor, the text should disappear as expected.
I don't think there is a way to edit the placeholder for now (might be wrong).

You can do it by overidding the Placeholder extensions via the extensions  props

@tiavina-mika
can you please show us an example for the Placeholder extension?

https://novel.sh/docs/guides/tailwind/extensions#default-extensions you can to that now

const PlaceholderExtension = Placeholder.configure({
  placeholder: ({ node }) => {
    if (node.type.name === "heading") {
      return `Heading ${node.attrs.level}`;
    }
    return "Press '/' for commands, or '++' for AI autocomplete...";
  },
  includeChildren: true,
});


Retrieve Users Domain (readable form)

Prepare the cache to retrieve the existing users domains in readable form. This should cover all domains (old and current).

Implemented in Precache. Closing issues.


[GRAV v1.7] Pages are unaccessible when in the Admin Panel

In Grav v1.7 pages are unaccessible from plugin when in the Admin Panel.
Code:
public function onPagesInitialized($event) {
    dump(array_keys($event['pages']->all()->toExtendedArray()));
}

This will display [] when you are in the Admin Panel.
This alternative also does not work:
public function onPagesInitialized() {
    dump(array_keys($this->grav['pages']->all()->toExtendedArray()));
}

More info can be found in this forum post.

my guess is you are use the wrong class, since you mention 1.7, my guess is your frontend is using legacy pages, and your admin is using flex pages so you get different results

Yes, that would make sense. However I still don't quite understand how I can use the correct class. In the plugin I don't think I use the use keyword with anything page related. Am I missing something?

@ricardo118 , I have copied the same plugin into two fresh installs of latest Grav 1.7 and 1.6.
When running $page = $event['pages']->find('/typography', true); in function onPagesInitialized() while running Admin plugin:

It returns a Page object in 1.6
It returns null in 1.7

@pamtbaau what plugin is it? and can you come to discord and ping me there? i can help you debug what's wrong

@ricardo118 , I have created a test plugin using devtools. I rather not come to discord. Visited a view times, but really dislike the place...
Here are the steps to reproduce:
Download latest Grav 1.6 + Grav 1.7
For each installation do the following:

$ bin/gpm install devtools
$ bin/plugin devtools newplugin
I named it 'test'
$ cd user/plugins/test
$ composer update
Changed codelines (using vscode)public function onPluginsInitialized()
{
    // Don't proceed if we are in the admin plugin
    if ($this->isAdmin()) {
        $this->enable([
            // Put your main events here
            'onPagesInitialized' => ['onPagesInitialized', 0],
        ]);
        return;
    }

    // Enable the main events we are interested in
    $this->enable([
        // Put your main events here
    ]);
}

public function onPagesInitialized($event) {
    $page = $event['pages']->find('/typography', true);
    dump($page);
}

Open Admin plugin

Results:

Grav 1.6: Dump of Page object of '/typography'
Grav 1.7: null

I recommend reading https://github.com/getgrav/grav/blob/1.7/UPGRADE-1.7.md

Ahh... Didn't look at the migration docs.
I presume you meant the section on breaking changes for Admin plugins?

BC BREAK Admin will not initialize frontend pages anymore, this has been done to greatly speed up Admin plugin.
Please call $grav['admin']->enablePages() or {% do admin.enablePages() %} if you need to access frontend pages. This call can be safely made multiple times.

The following works in Grav 1.7.
public function onPagesInitialized($event) {
    $page = $event['pages']->find('/typography', true);   // null
    $this->grav['admin']->enablePages();
    $page = $event['pages']->find('/typography', true);   // Correct Page object
}

Thanks @mahagr !

You don't need the first line, but I suppose you added it for an example. If you also want to make the plugin compatible for Grav 1.6, you can check if the enablePages() method exists.

Yes, the first line was only to show the effect of enablePages().
Btw. extensive migration list... Also many breaking changes. I wonder how this will workout with plugins. I have the impression that many are very old if not abandoned. Are there any thoughts on this?

Most of the breaking changes are really not common at all, or some special cases or very old approaches. Or things that were not really working and fixing them changed the behaviour.

The issue has been resolved. Thanks everyone for help.


fix: Add Send trait constraint for async_decrypt_layer

This trait constraint is needed when the caller of this function wants to transfer it between threads safely
Related error occurs upon the newest image-rs code when bring in async like https://github.com/confidential-containers/image-rs/pull/108

The error reported seems not related to this PR.

The error reported seems not related to this PR.

Need to update the ci action to use "apt-get update && apt-get install xxxx".


Better support for Section[5]/strings/vtables

BrawlBox/BrawlCrate has support for all the vtable stuff that goes in Section[5]. Right now, it's just treated like raw words with some relocations in there. Somehow, it would be nice if the class and function name strings were actually parsed and easier to see / search through / edit. (Granted, I don't know very well how these really work or how they're supposed to be used in the first place lol.)

The vtables are just that, actually. They are arrays of relocations to functions. There are no function name strings, only the name of the class the vtable belongs to. The data that is actually at those addresses in Section[5] is empty and the way we know what the function names are is due to the map files on the korean brawl disc.
I am probably going to be disassembling the string names for the classes though. It reduces the size of the resulting text file by a lot, and they are part of the custom Runtime Type Information format that the CodeWarrior compiler generates. They are useful for identifying what class each vtable belongs to.

Sorry, right - the strings are just class names, not function names. I have seen them being used to make dynamic_cast work. Being able to add new entries seems like it's necessary for properly porting some functions from a module to another.

The tool uses gnu as under the hood so if you port anything over and use .asciz or .string it will assemble correctly. Editing a vtable entry should be just editing the relocation tag at the end of the line to point to the symbol for the function you want to point to. Similarly, adding a new entry to the vtable would just be adding a .4byte 0x00000000 line and relocation tag.
Hopefully in the near future I'll be adding some checks to disassemble ascii data automatically. This way it won't need to be done manually anymore.
Aside from disassembling the strings properly, any abstraction layer or syntax on top of the vtable structure would be outside the scope of the tool. Most games have RTTI disabled and don't follow Brawl's conventions anyways, so it would become game specific code.
This does, however, give me some neat ideas about a VS Code extension for working with the output of this tool. Perhaps one which adds tooltips, auto-complete, navigation, and code snippets for easily creating CodeWarrior vtables.

Support for auto-disassembly of strings was added in 19fe75d3994ccddc86535406446abefab219699f.
Note that there is still the chance of false positives in this, but they should fairly unlikely. I've only run into it on custom rels like bx_fighter.rel for the BrawlEx. Even in the case of a false positive, the data should still compile and match even if it's represented as string in the source.


manytomany DataReference related changes added

@4manasa There are merge conflicts that need to be resolved.

yeah, fixing the conflicts.


Announce in Slack if automerge fails because PR is not mergeable

Fixing failed workflow run: https://github.com/Expensify/Expensify.cash/runs/2143920797?check_suite_focus=true
The mergeable_state of a PR is mix of whether it is mergeable and has all its PR checks completed and passed. While checks are still running, the mergeable_state is unstable. Both mergeable and mergeable_state are determined asynchronously in Github using background jobs.
So we have a custom Javascript action to poll continually until the mergeability of a PR is settled. This PR adds a slack announcement in the case that an automerge PR is created that's not mergeable (should not happen, hence the slack announcement). We also could poll and wait for the mergeable_state to be settled, but frankly we don't care about that field, since we skip all the checks on the automerge PRs anyways. So this PR also removes the checks we had in place for mergeable_state, which caused the automerge workflow to silently fail unnecessarily.

Merging to get deploys working!

@AndrewGable looks like this was merged without passing tests. Please add a note explaining why this was done or remove the Emergency label if this is not an emergency.

Worked for the master PR https://github.com/Expensify/Expensify.cash/actions/runs/666173020
Worked for the staging PR https://github.com/Expensify/Expensify.cash/runs/2144212961?check_suite_focus=true

🎉


Authentication Fails

Authentication fails  with passwords containing special characters like %,$ ... etc.

I may be doing something incorrectly, but I've tested this using a password containing =, %, and $. I am able to authenticate and log into my local instance just fine.
Are you able to provide any additional detail?

Passwords I've tried is with ~%*_ , using SCRAM-SHA-1 authenticantion.
A solution that I've found is to use encodeURIComponent and uri_decode_auth when doing a new connection which is supported by the driver. See here
newDb.auth.username = encodeURIComponent(options.auth.username);
newDb.auth.password = encodeURIComponent(options.auth.password);
client.connect(this.connectionString, {uri_decode_auth: true}, (err, database) => {});

Have you opened a PR for your solution? If not do you mind if I do? I will credit you with the fix.

 The only thing is that this change wont for previously saved passwords. must re-validate this.


Question: How to use path parameters?

Is it possible to define and use path parameters in custom find methods?
repository.find('/articles/{article}/editors',{article:1}).then(editors=>{

});

No, it is not. You're looking for criteria, findPath, or direct access to aurelia-api.
But to me it sounds like you want to look for /editor?article=1

Ok, the real issue is, that findPath() is undocumented here https://aurelia-orm.spoonx.org/api_repository.html

@mreiche I agree. That's a problem....


DTD-2721 Update stub with mandatory fields for ETMP VATC/PAYE

Deleted changes made to SA and added AddressLine3 and 4 to first file in PAYE and VATC.


Add HUGINN_AGENT_SPEC_RUNNER helper

Changes:

When HUGINN_AGENT_SPEC_RUNNER is set - load the huginn_agent.gemspec. This allows the gemspec development dependencies to be loaded when running rake spec from huginn_agent

Fixes:
https://github.com/cantino/huginn_agent/issues/13
Blocks:
https://github.com/cantino/huginn_agent/pull/15

Closing in favor of:
https://github.com/cantino/huginn_agent/pull/17


Can't launch VSCode after latest update

VSCode Version: 1.37.0
OS Version: 10.0.17763

Steps to Reproduce:

run code --version

Result:
Fatal error in , line 0
Check failed: (i_isolate->snapshot_blob()) == nullptr.
FailureMessage Object: 0000004D6D4FF578
I normally launch VSCode using a Windows shortcut on my start menu, however after not launching I tried with powershell and it still didn't work.

Can you try to reproduce with our nightly insider builds? You can give our preview releases a try from: https://code.visualstudio.com/insiders/

I have the same problem with 1.37.0. Rolling back to 1.36.1 solves it. The nightly insider also works.

1.37.0 started working now too. I use the portable zip version. I replace my previous installation each time I upgrade, keeping the data/ folder. After some back and forth with 1.36.1, insider, and 1.37.0, it started working. This time, I also started up the unzipped 1.37.0 before moving it to replace the old installation, and without the old data/ folder. It still works after replacing the old installation while keeping the data/ folder. I don't know which step solved it.

Update: I got VSCode working again thanks to @Lajnold 's suggestions. I'm going to list my steps below in case anyone else is having the same problem.

I downloaded the 64bit .zip version from https://code.visualstudio.com/Download

I made a new folder for VSCode (mine was under C:/Users/user/AppData/Local/Programs/) and extracted the .zip file (without touching the old folder)

Ran Code.exe and it worked!

Keep in mind that it does indeed keep your settings/extensions


[DOCFIX] Update the Configuring-Alluxio-with-S3 page of Chinese Doc.

Update the Configuring-Alluxio-with-S3 page of Chinese Doc.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/9555/
Test PASSed.

LGTM, thank you for the translation!

LGTM


Feature Request: Insert Task List Renderer to show future tasks

Hi,
First, thanks for this great plugin.
I would like to suggest that the Insert Task List Renderer option could include the tasks scheduled for the next few days. It could be as simple as like the overdue tasks, but showing in the end of the list and with a different color, maybe blue.
If possible, the amount of days ahead to show could be a configuration option. If not, I suggest that it show the next 7 days.

Thanks for the suggestion, there are indeed plans to show tasks in the task list for a specified number of days in the future.


HSEARCH-4948 Vector search using the Lucene backend

https://hibernate.atlassian.net/browse/HSEARCH-4948
Hey Yoann,
I've put together a few things that would allow storing vector fields and thought that it would be a good time to check with you before moving forward with exploring the searching part...
I went with the property names for a vector field that are closer to the Lucene attributes rather than what Elasticsearch/OpenSearch are using (m/ef_construction do not sound like a nice option for method names 😆; hence it is  maxConnections and beamWidth instead).
With that said, note the ScopedCodec -- it turns out the only way to actually pass these two values is through a codec that is a part of the IndexWriterConfig.  I was hesitant about where to put knnVectorFormat but as it is somewhat codec-related I've put it there for now...
Let me know if you think we'd need something additional in the VectorField and what do you think about the naming of the public API side ... 😃

oook, this should now build all of the modules and have some docs 😃
I'll move it out of the draft and let me know if there's something missing for this PR 😃

I've added a few more commits for the bridges/binders and some checks. you'll find them after the commit for step initiatiors 😃. Also eclipse compiler was complaining so I had some more changes applied to the TCK tests

Started a build for all envs here https://ci.hibernate.org/blue/organizations/jenkins/hibernate-search/detail/PR-3814/26/pipeline to make sure there will be no surprises once merged (I've tried to build with JDK 11 and had to update some of the new tests 😔)

The cockroachdb failure is definitely a flaky test. As for the eclipse compiler thing... I don't know, it probably got cancelled when the other one failed. Let's merge and see what happens on main.
Thanks for all this work!

As for the eclipse compiler thing

I've just tried it locally and it passed, so hopefully no surprises there 🤞🏻