How many epochs should be set for resnet on ImageNet?

I'm confused by the epochs in resnet example and what is reported on resnet paper:

imagenet-resnet.py

In the README of imagenet-resnet.py, you report the command line to run resnet is as below:
./imagenet-resnet.py --data /path/to/original/ILSVRC --gpu 0,1,2,3 -d 50 [--mode resnet/preact/se]
Also, you provide your performance figure. From the figure, we can see the total epoch is 120.

As you explained in #353 , one iteration update weights with the data size of TOTAL_BATCH_SIZE. As explained in the document:
In each iteration all GPUs (all replicates of the model) will take tensors from the InputSource, instead of taking one for all and split.

It means one epoch the trainer will go through the whole dataset four times as there are 4 gpus. When it runs for 120 epochs, I think it means it will go through the dataset for 480 times.

Resnet paper

As reported in resnet paper, the training runs for 600000 iterations. As the total training dataset size of imagenet is 1281167 and batch size is 256, the trainer needs to run 1281167/256=5004 iteration to go through the dataset once. So it means the training go through the dataset 600000/5004=120 times.

If my understanding doesn't go wrong, it seems the multiple gpu training will go through the dataset 4 times greater than what is reported in resnet paper.

Do I miss something when I get 480 times by calculation?

Do you know what the iteration means in resnet paper? Should I also count #gpu into one iteration?

TOTAL_BATCH_SIZE is always 256 no matter how many GPUs you use. So one epoch is always one epoch. The number of GPUs only affects how many samples per GPU.
The original resnet paper means the same thing. It may differ a little bit (i.e. 5004 vs 5000) but that's negligible.

Do you mean in one epoch, 4 GPUs will go through the dataset one time instead of 4 times?

On Dec 8, 2017, at 23:19, Yuxin Wu<EMAIL_ADDRESS>wrote:
TOTAL_BATCH_SIZE is always 256 no matter how many GPUs you use. So one epoch is always one epoch. The number of GPUs only affects how many samples per GPU.
The original resnet paper means the same thing. It may differ a little bit (i.e. 5004 vs 5000) but that's negligible.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub https://github.com/ppwwyyxx/tensorpack/issues/544#issuecomment-350427331, or mute the thread https://github.com/notifications/unsubscribe-auth/AFNYNMBeBa6zSg7k2S7IlTdl3yw6bIyBks5s-iZlgaJpZM4Q76Pw.

Yes.

When I set the TOTAL_BATCH_SIZE = 192, the progress bar will show something like 4908/26690. What does that mean? Is this the progress of iteration?
It seems to make sense that 26690 = 1281167/48 (the size of training is 1281167).

On Dec 8, 2017, at 23:54, Yuxin Wu<EMAIL_ADDRESS>wrote:
Yes.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub https://github.com/ppwwyyxx/tensorpack/issues/544#issuecomment-350428750, or mute the thread https://github.com/notifications/unsubscribe-auth/AFNYNNq9bEYK4kN-oCP9Xw-4RaQZxwwdks5s-i6NgaJpZM4Q76Pw.

Please always paste your observation instead of describing it. I do not understand what you observed.
As far as I know you will not see anything like 4908/26690 if changing TOTAL_BATCH_SIZE is the only thing you did.

You will NOT see anything like this if you're using the ResNet example in this repo and only changed the batch size. Please confirm what you did, including the changes you made and command you run.

Closing because information about the problem was not provided. Reopen if you still have questions.


Adding API for Alert Status

This pr will enable to view/edit/delete all alerts-statuses.
The api support watching and interacting (like a ticketing system) with all of the current alerts on the cloud/infra/container providers.
As an operation team using manageiq it will enable to get alerts from the current providers and also comment/ack/assign those alerts to other miq-users, this will enable easier alerts solving.
cc @simon3z @zgalor

@miq-bot add_label wip,api

@miq-bot add_label wip,api,enhancement

@abellotti @imtayadeway please review guys.

@alongoldboim it's hard for me to get a high-level picture of what problems this solves. Are you able to update your PR description with a user story/stories, along the lines of "As a , in order to , so that ". It will then be easier to see to what extent the implementation meets/surpasses the requirements. Thanks!

@imtayadeway Please see #11962 which is the consumer of this API.

@jeff-phillips-18 @alongoldboim I would prefer the description of this PR to be updated with the why for this change, as it will save the next person reviewing to have to dig through the details. I suggested a user story for this as I find them to be helpful in clarifying these details, but any description clarifying the why aspect will do.

@imtayadeway I have edited the description.

@imtayadeway @abellotti when doing "api/providers/1/alert_statuses"
"undefined method `id' for #<Hash:0x007fd3a1566c98>"

I think the api expects a resource with id, should i use openstruct maybe instead of a regular hash, or is there an easier way i missing?

@miq-bot assign abellotti

LGTM :+1: up to @abellotti to review/merge
cc @chessbyte

Also, Rubocop warnings above.  Thank you.


Mask reuse in home setting?

Mask question from someon: "I'm in isolation w/in isolation, separated at home from my mildly symptomatic family while we await test results. We have just a few masks. If virus can only survive for a certain time, can I re-use mask after several days, disinfecting the little metal strip?"
c/o https://twitter.com/JeffSharlet/status/1239641846613147651

Need someone to +1 this

We should pull best practices for mask use and reuse for different types, and what the tradeoffs are to help people make good decisions.   A few topics to consider...

How to take on / off masks - dont touch outside
Where to store masks that you are reusing (away from kids, pets, traffic, etc..)
How to fit and size masks - this is hard when not trained
What surgical masks are good for (Sick People)
Not to hoard N95 masks - health providers need these.

Here's some research I did on mask reuse:
From: https://www.facebook.com/thismattbell/posts/10159622229081038
Since mask supply is limited, there are questions about mask reuse. There are conflicting reports about whether it is safe to sterilize and reuse masks. [1] [4] Almost everyone recommends discarding masks once used in a high-risk setting (eg a hospital, or contact with a person with COVID-19). However, reuse after low-risk use may be possible. Some sources claim that improvised fabric or surgical masks can be sterilized in boiling water and then dried, disinfected with UVC light, or left out in the sunlight for several days. [0] [1] N95 masks need to stay dry. An old study from SARS-CoV suggests that temperatures of 60C / 140F for 30 minutes will destroy a different coronavirus. [2] There is no analogous study for this coronavirus yet, but it’s likely its temperature tolerance is similar. N95 masks appear to only mildly be damaged by this level of heat. [3] Thus, it’s possible you could “bake” your used N95 masks at low heat in the oven (eg 180F) to sterilize them without damaging them. I’ve tested this on the 3M brand N95 masks we have and it appears to work.
The case for masks
[0] https://www.inkstonenews.com/health/coronavirus-health-researchers-urge-global-authorities-consider-broadening-use-face-masks/article/3074453
Washing surgical masks:
[1] https://focustaiwan.tw/society/202002020012
Heat for inactivation of the original SARS coronavirus:
[2] https://www.ncbi.nlm.nih.gov/pubmed/15118911
Mask safety during sanitization
[3] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2781738/
Claims of mask steaming as ineffective:[4] https://factcheck.afp.com/novel-coronavirus-health-experts-warn-against-steaming-face-masks-reuse-after-misinformation-chinese

Possibility of fomite transmission can't be ignored, especially now. 3 days would be the lowest limit I am comfortable recommending. 9 days, see https://github.com/flattenthecurve/guide/issues/34

One thing that the Lancet article doesn't directly address is the issue of mask destigmatization. If the messaging is that only sick people should wear masks, the assumption will be that people wearing masks are sick and vulnerable to prejudice. By normalizing (home-made) mask use, we could improve outcomes for everyone, not only because those who are symptomatic will feel free to use them, but also because asymptomatic transmission is clearly such a big problem.

I spent a lot of time researching mask reuse.  I put this Medium post up yesterday, and it has a significant section on mask reuse:  https://medium.com/@llebttam_45762/masks-work-wear-them-b450ac9f3b7d
I'd be happy to turn any portion of this into a post for flattenthecurve if you'd like.  Just let me know the scope and length and I can make it happen.

US health services group "Deaconess Health Systems" shares a mask design PDF and Youtube Video - Mar 20, 2020
US health services group Providence shares a mask design video - Mar 21, 2020 (part of their 100 million mask initiative)

A 2019 study on masks: Modeling the Effectiveness of Respiratory Protective Devices in Reducing Influenza Outbreak (Listed on the USA's "National Center for Biotechnology Information" site)
From that study's "Results" section: The level of compliance was assumed to be the same for the susceptible and infected populations. [...] A 20% compliance rate [of surgical or better masks] cuts the infection prevalence roughly in half and delays the peak of the epidemic to around day 25. For 50% compliance [of surgical or better masks] reduce the prevalence to less than about 5%. At 80% compliance, the infection prevalence is negligible where the maximum is roughly 5%.
Anyone still think that FlattenTheCurve.com should not move to "masks work" language?
Qualified as surgical masks != N95 masks != cloth or home made masks.


python3Packages.tinysegmenter: init at 0.3

Description of changes
Very compact Japanese tokenizer
https://tinysegmenter.tuxfamily.org

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[x] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

@ofborg build python311Packages.tinysegmenter python312Packages.tinysegmenter
I'm concerned about distutils.
https://git.tuxfamily.org/tinysegmente/tinysegmenter.git/tree/setup.py


I Would Like To Contribute Some Smothering Ideas To The App (For Desktop as well as Mobile site), Please Allow Me...

Hey There, I feel like I have Some Great Ideas that I think I can implement on this app, and correct some of the documention error that i found. So is it possible for you to kindly assign this project to me, so that i can help.

I would like to work on this issue, please assign this issue to me under GSSoC'23.


Update InstallNuKeeperDotnetTool.bat

pack artifacts to root directory instead of NuKeeper/artifacts
:sparkles: What kind of change does this PR introduce? (Bug fix, feature, docs update...)
:arrow_heading_down: What is the current behavior?
:new: What is the new behavior (if this is a feature change)?
:boom: Does this PR introduce a breaking change?
:bug: Recommendations for testing
:memo: Links to relevant issues/docs
:thinking: Checklist before submitting

[x] All projects build
[x] Relevant documentation was updated

Thanks

argh travis is failing, @AnthonySteele  can we deprecate travis?

Can do, we're confident that Azure can do it. I will have a poke around to find where to remove the integration.

Gone!

Do we still want this one?


npm build for Sugarizer has deprecated dependencies

Module dependencies for Sugarizer has deprecated dependencies. This is shown in the npm install console log. We can probably work out the current maintained versions of the dependencies @llaske ?
Some are more critical and might expose RegEx DoS vulnerabilities as well.

@utkarsh-raj what's your node.js version?

NodeJS v8.15.0
npm v6.4.1
Does the issue have to be related to the client environment ?

I figured it. To log the solution for reference-
This is probably that v1.0 does not support NodeJS v8.x and above. Downgrading to v6.x solves the fibers build issues. Also I noticed that this deprecation issues need to be addressed for v1.1, if it has not been done already. Since the deprecated modules will not conflict or show warnings with older NodeJS versions, but the codebase needs to incorporate the maintained modules.
@llaske thoughts ?

For Sugarizer App, node.js is need only on dev environment (specifically to minify files) and to generate the electron version for Windows/Linux/MacOS.
So it's not a critical issue but it should be fixed for sure.
BTW fixing the issue is not only upgrading deprecated libraries, it means carefully testing on all environment.

@llaske Will require a bit of work for the testing. I am on it :)

It's fixed in the v1.4.0


Need support for FaqGen example in GMC

Need help to extend support for FAQGen example pipeline in GMC.

Hi @hteeyeoh, thanks for your requirement, basically, the helm chart should support the FaqGen example first, then GMC will support later, so I also @yongfengdu and @lianhao here as well.

Thanks @zhlsunshine . Appreciate that if @yongfengdu or @lianhao can give ETA on this if possible so that our team can plan accordingly. Thank you.

llm-faqgen-tgi support is proposed in PR #436. @zhlsunshine, you may proceeds with GMC planning.


Pin elasticsearch to 7.0.0 in requirements.txt

in accordance with fix_497 , the pin seems to be missing in the requirements.txt

Reflected in elastalert2


[Bug]: Edit the PL object so that the PL content can be modified during initialization

ODC version
421
OB version
all mysql schema
What happened?
When editing PL, the PL contents can be modified during initialization
What did you expect to happen?
The PL of the mysql schema can not be edited
How can we reproduce it (as minimally and precisely as possible)?
Trigger the PL editing function to modify the PL content during initialization
Anything else we need to know?
Cloud

Fixed In Commit cbdee53be86e7702447bb38c18ae3066308ea4c4
------
晓康
Wed Sep 6 15:46:29 2023 +0800

test passed


fields: XPaths should not select codelists

The XPath-expressions of some fields pointed to their codelist and not to their value.

Thanks for your feedback.
I'm not sure I understand the issue: those fields have XPaths that point to an attribute because that's where the information is.
For example:
"BT-26(a)-Lot" should indicate the type of classification used (CPV), and this is indicated by the value of the "listName" attribute.
Removing "/@listName" from the XPath of "BT-26(a)-Lot" means it then has exactly the same XPath as "BT-263-Lot" (the value in the CPV classification).

Oops, I'm sorry.
I was indeed looking for the wrong BTs, didn't release separate fields for attribute and value.


rename wifi module

The wifi module is actually not only wifi and should be renamed network or something similar.

This is done now and merged to main. I think there remains a good amount of non wifi code in wifi.py and there is likely a lot of dbus network manager code duplication that could be refactored. But this will be in other issues.


Replace scalafiddle reference with scastie

Seems like the good-old (dead now) ScalaFiddle redirects to https://bee-line.dk/ for some reason...
https://scalameta.org/docs/trees/scalafiddle.html shows an iframe of this site: https://bee-line.dk/ :sweat:
This PR replaces it with a Scastie playground.

should we do the same for scalafmt? https://scalameta.org/scalafmt/docs/configuration.html#aligntokens

should we do the same for scalafmt? https://scalameta.org/scalafmt/docs/configuration.html#aligntokens

https://github.com/scalameta/scalafmt/pull/3843


Update README.mkd

Hey, I just changed the windows install command in the readme file

I don't have a windows machine to test this on, anyone else wanna take a second look?

I just pulled it down the other day on windows. Might have a chance to try it today or tomorrow

Also edited the motd

I'm going to close this PR so the MOTD change can be made separately from the, now unnecessary, README update


fix https://github.com/zimfw/completion/issues/13

Amazing that you found the exact fix. Good catch!
I still cannot reproduce the issue in my macOS even with the super detailed test that you've posted above. Maybe you have a path or file with special characters that I don't have.
Does it also work if you use LC_CTYPE=C instead of LC_ALL=C?

Oh okay, good point. I just have second thoughts on merging this because we're using the same code that compinint uses to collate the completions:
https://github.com/zsh-users/zsh/blob/42640b26136cbc1e35df08210028a3c3f41161f9/Completion/compinit#L523
I'm wondering if the right solution would not be to have separate compdump files for each LANG (or LC_ALL):
zstyle ':zim:completion' dumpfile "~/.zcompdump.${LANG}"

since it seems compinit would generate different files in each case.

Okay, if that has only to do with the order -- although we're not using sort in our code -- then I think I will not affect how compinit works, but will affect how the module code works because the order matters to the module logic.

The compdump file is different when the locale orders the _ character differently as you've shown:
% for lc (C cs_CZ.ISO8859-2) LC_ALL=${lc} compinit -C -d /tmp/zcompdump.${lc}
% diff -q /tmp/zcompdump.{C,cs_CZ.ISO8859-2}
Files /tmp/zcompdump.C and /tmp/zcompdump.cs_CZ.ISO8859-2 differ

It's just a matter of order inside the file, yes.


Consider immutable JS

Or immer.js
But do we need it?  @ngrx/entity does NOT use an immutable library to improve performance. I think they have something cooking with freeze to prevent mutation.

Sorry to bring back an old thread but,
Is it good practice to use an immutable library like immutable.js in @ngrx then?
Surely it's better for guaranteed safety... otherwise a lot of discipline is required to ensure immutability (which gets harder on bigger teams)
Or do you guys feel its unnecessary and plain JS objects are sufficient?


WIP: Adds upload documents screen

This relies on the https://github.com/codeforamerica/form-flow/pull/74 PR/Branch
This adds in the file upload page in the figma drawings.
This tweaks the design of the "How to add documents" page as well, to fix some formatting.

Still needs:

icon in the reveal header
I'm finished uploading button should not show up until a file has been uploaded
needs the success screen, which is not listed in the ticket, but should be.


and  break the parsing.

Hi,
If I make my markup as so:

Websites are more than just pages.

Let us help you bring yours to life.

With this initialization method:
$('.mejor').textillate({ in: { effect: 'rollIn' } });

The effect is as expected:

However, if I insert a break into the h2 element, like so:

Websites are more than just pages.

Let us help you bring yours to life.

It breaks the parsing, resulting in this:

Since the underlying technology can handle these
 elements, I thought you should know. Otherwise great tool! Thanks.

Did this ever get added?
I've just tried  in my HTML but it showed it as  rather than breaking the line.


Change default tolerances

This PR changes the default tolerance to check equilibrium.
It introduces tol_force=0.01 and tol_angle=0.1 which will be used to check maximum angle length (or force) deviation and maximum angle deviation since these values have very different sensitivities.
This PR is built upon #127

I think these tolerance settings should live in the global 'IGS' settings in compas-IGS. Refer to https://github.com/BlockResearchGroup/compas-RV2/blob/main/src/compas_rv2/ui/Rhino/RV2/dev/RV2init_cmd.py.

Yes, I agree. but we need a place to enter it as an argument of the function... what's what this PR does. The definition of the value for this tolerance will come indeed from IGS

After discussion with @juney-lee this PR was modified and now the bidirectional function update_diagrams_from_constraints does not check for the equilibrium at each iteration, it just runs max_iter times


Update to Angular 12

Issue type
I'm submitting a ...  (check one with "x")

[ ] bug report
[x] feature request
[ ] question about the decisions made in the repository

Issue description
Hello, I'd like to know if you are working to update to Angular 12.
Thanks in advance.

Seems like updating to angular 12 will break the responsive design of the admin dashboard. Would be nice if we could get a fix.

Any update on this?

Any update?

looks like new angular version change too quickly

Guys, open your eyes, it makes a long time that this project is on angular 12..

Guys, open your eyes, it makes a long time that this project is on angular 12..

you are correct but the package with the backend from the website is still the old version. guess the questions here are not responsed frequently. my bad.


Origin Server does not get added properly as part of build

Observed 3x so far, Origin server shows up in Origin pool under LB mgmt in XC Console, but does not show up as an Origin Server under the HTTP LB Dashboard in XC Console. Removing the existing node from origin, and re-adding manually fixes the issue. Behavior is repeatable in my tests. No errors/deployment issues with module. Only indication of improper deployment is LB FQDN throws a 503 error.

line 90 of main.tf had incorrect namespace setting in my tf cache. Source is correct in the repo, so no changes needed


Remove "monticello username" from login dialog

What the title says.
Also consider not defaulting to localhost - leaving it empty will prompt the user to fill it in.

Looks like Jade records the login information in the registry as suspected. If Jade has never been installed on a machine, my theory is that the login dialog will open up with a blank host. (or some sensible default). Thereafter, it will save your last login to the registry.
Does this suffice?
Here's the method for reference:
JadeLoginShell>>defaultPath
    | regKey appData |
    regKey := RegKey userRoot at: 'Software\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders' ifAbsent: [nil].
    appData := regKey valueAt: 'AppData'.
    (appData endsWith: ' Dat') ifTrue: [appData := appData , 'a'].          "Wine seems to cut off a character"
    ^appData , '\JadeDefaultConnection.gss'.

~

(oh, I'll have the Monticello name removed at the next .exe drop)


Default appendElement

If there isn't a way to set a default appendElement, that would be a great feature. It is probably pretty common that if you define a particular appendElement element that you will always want to append to that element.

By default modal is appended into body. Change body var for your need line in angular-modal-service.js:46
//  Get the body of the document, we'll add the modal to this.
var body = angular.element( $document[0].body );

@wojo1206 That would work, but then I would need our own fork of angular-modal-service and couldn't keep it updated with bower.

This is available, see appendElement at: https://github.com/dwmkerr/angular-modal-service#showmodal-options


Automator: update proxy@master in istio/istio@master

Generated by Automator - 2021-11-09T19:27:27+00:00

@istio-testing: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-security-k8s-tests_istio
75919567049b0a6fe63a20062a7011efd0923083
link
true
/test integ-security-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test integ-security-k8s-tests_istio

@istio-testing: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-security-k8s-tests_istio
94ca3f3518790fea4ceda5ecede4b3e6fa309f5d
link
true
/test integ-security-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test integ-security-k8s-tests_istio

@istio-testing: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-security-k8s-tests_istio
94ca3f3518790fea4ceda5ecede4b3e6fa309f5d
link
true
/test integ-security-k8s-tests_istio

integ-security-multicluster-tests_istio
94ca3f3518790fea4ceda5ecede4b3e6fa309f5d
link
true
/test integ-security-multicluster-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-security-istiodremote-tests_istio
94ca3f3518790fea4ceda5ecede4b3e6fa309f5d
link
unknown
/test integ-security-istiodremote-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test integ-security-istiodremote-tests_istio


Feature request - performance metrics triggered auto-capture

Description

In complex scenes, the performance metrics may vary rapidly and it is quite difficult to capture the frame we are interested in manually.
It would be great if Renderdoc can capture automatically when a performance metric like draw call hits the user set bar.

UI: add an option to "Caption Options" with an on/off button like "Auto-capture". With "Auto-capture" on, a user should be able to select from a drop list like "Metrics Type", then set the related value.
renderdoc: If the feature is on, should be able to calculate related Metrics in a frame, such as a number of draw calls. When the condition is fulfilled, start the capture process.
Environment

RenderDoc version: v1.13
Operating System: Win10
Graphics API: GLES 3.2

RenderDoc is not a profiler so supporting performance analysis workflows is not currently a goal of the project. This could maybe be accomplished manually using the in-application API to trigger a capture on a custom condition, but note that it would be off by one frame since it's not possible to retroactively capture a frame.


Switches to launch with main class vs PropertiesLauncher when possible

After researching before and after startup times with Spring Boot 2.1
vs 2.2, the largest win for startup time, regardless of version, is to
not use PropertiesLauncher.
Here is basic startup time before and after with the 2.17.2 version of
Zipkin:
Before (using PropertiesLauncher)
Started ZipkinServer in 3.641 seconds (JVM running for 5.009)

After (running the main class)
Started ZipkinServer in 3.267 seconds (JVM running for 4.31)

Note: even before this change we extracted Zipkin's jar first, so the
entire gains here are only from not using PropertiesLauncher.

@anuraaga FYI: probably we'll want to do the same in the other images built, as it is still nicer to start faster even from master tag.


Inconsistency in visualizer lifecycle; should it be changed?

Like the testing issues, the documentation issues are great for surfacing underlying problems. Responsive to #46, I am designing a "visualizer lifecycle diagram" (along the lines of the Vue component lifecycle diagram). That exercise has uncovered the following inconsistency:

We document the checkParameters() method of a (P5)Visualizer as being able to rely on, and use, the sequence .first and .last properties in validating the parameters of the visualizer, and indeed, some of our Visualizer classes (e.g. Turtle) do so. Yet when the sequence changes, the update pathway only goes back to presketch(); if no parameters specific to the visualizer have changed, the visualizer checkParameters() is not called again. So whatever validations were done on the sequence extents may now be in a failed state. This mismatch is even called out as a bug in the comments in Turtle.ts.

Should we modify the update pathway on sequence change for a Visualizer to include calling checkParameters()? Or if not, how should the above inconsistency be resolved? I do not think the inconsistency should be allowed to remain.

Is there any reason not to modify the update pathway as you suggest?

I can't think of any reason other than it's yet another change and will have various repercussions we need to track down and likely fix. But in the long run, it definitely seems like a better scheme. So we could just go for it now while everything is in turmoil anyway ;-)

Sounds like updating the pathway makes sense then?

OK, renamed the issue accordingly.

Implemented in ui2 by #486. Closing.


Be more lenient with jwplayer regex: Allow Brackets '(' and ')' in config

Please follow the guide below

You will be asked some questions, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your pull request (like that [x])
Use Preview tab to see how your pull request will actually look like

Before submitting a pull request make sure you have:

[x] At least skimmed through adding new extractor tutorial and youtube-dl coding conventions sections
[x] Searched the bugtracker for similar pull requests
[ ] Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

[x] I am the original author of this code and I am willing to release it under Unlicense
[ ] I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

[x] Bug fix
[ ] Improvement
[ ] New extractor
[ ] New feature

Description of your pull request and other information
This website contains a jwplayer video: https://mediathek.hhu.de/watch/73f80093-d3d4-489a-9e2f-2b0bf6a3cae3
The generic extractor gives this info though:
 youtube-dl -vF https://mediathek.hhu.de/watch/73f80093-d3d4-489a-9e2f-2b0bf6a3cae3
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-vF', 'https://mediathek.hhu.de/watch/73f80093-d3d4-489a-9e2f-2b0bf6a3cae3']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dl version 2020.11.29
[debug] Python version 3.9.0 (CPython) - Windows-10-10.0.18362-SP0
[debug] exe versions: ffmpeg n4.3.1-20-g8a2acdc6da, ffprobe n4.3.1-20-g8a2acdc6da
[debug] Proxy map: {}
[generic] 73f80093-d3d4-489a-9e2f-2b0bf6a3cae3: Requesting header
WARNING: Falling back on generic information extractor.
[generic] 73f80093-d3d4-489a-9e2f-2b0bf6a3cae3: Downloading webpage
[generic] 73f80093-d3d4-489a-9e2f-2b0bf6a3cae3: Extracting information
ERROR: Unsupported URL: https://mediathek.hhu.de/watch/73f80093-d3d4-489a-9e2f-2b0bf6a3cae3
Traceback (most recent call last):
  File "c:\program files\python39\lib\site-packages\youtube_dl\YoutubeDL.py", line 803, in wrapper
    return func(self, *args, **kwargs)
  File "c:\program files\python39\lib\site-packages\youtube_dl\YoutubeDL.py", line 824, in __extract_info
    ie_result = ie.extract(url)
  File "c:\program files\python39\lib\site-packages\youtube_dl\extractor\common.py", line 532, in extract
    ie_result = self._real_extract(url)
  File "c:\program files\python39\lib\site-packages\youtube_dl\extractor\generic.py", line 3381, in _real_extract
    raise UnsupportedError(url)
youtube_dl.utils.UnsupportedError: Unsupported URL: https://mediathek.hhu.de/watch/73f80093-d3d4-489a-9e2f-2b0bf6a3cae3

This is caused by a peculiar config (from the source):
                           logo: {
                                file: '/images/watermark-hhu-2.png',
                                link: encodeURI('https://mediathek.hhu.de/watch/73f80093-d3d4-489a-9e2f-2b0bf6a3cae3/'),
                                position: 'top-right',
                                hide: true,
                                margin: 10
                            },

As you can see there is call of the JS function encodeURI(...)
This did not work before, as the RegEx excluded round brackets. I added that capability.
Old regex: https://regexr.com/5hdfq
My new regex: https://regexr.com/5hdgi
It will match everything until the first curly brace '}' that is followed by a ')' (with some optional whitespaces in between).

This is a slightly better version than #21132
All to allow #22660

Add tests.


check-file-exists-in-print-profile

I tried to run the command "LuaCacheProfile" and I was getting this error
Error executing Lua callback: ...share/nvim/lazy/impatient.nvim/lua/impatient/profile.lua:200: attempt to index a nil value
stack traceback:
        ...share/nvim/lazy/impatient.nvim/lua/impatient/profile.lua:200: in function 'print_profile'
        ...share/nvim/lazy/impatient.nvim/lua/impatient/profile.lua:327: in function <...share/nvim/lazy/impatient.nvim/lua/impatient/profile.lua:326>

I didn't see much through the whole code, but the one causing this problem was accessing a file that doesn't exists. so this pull request checks if the file exists or not before accessing the "size" of the file.

I am making this pull request to bring the issue

In nvim 0.9 you no longer need this plugin. Instead run vim.loader.enable().


shape_invariants "None" for all loop variables for tf 1.15.x

OS Platform and Distribution : macOS Catalina 10.15.3
TensorFlow installed from : binary
TensorFlow version : 1.15.0
Python version: 3.7.3
When I try below
import tensorflow as tf

tf.enable_eager_execution()

def tile_nd_ragged2(a, b):
  # Need a sentinel, otherwise it's hard to give it the initial shape we need.
  # We'll drop the sentinel at the end.
  acc = tf.ragged.constant([[[]]], dtype=a.dtype)

  # Work one row at a time...
  for i1 in range(len(a.nested_row_lengths()[0])):    # Should be able to write `for a1, b1 in zip(a, b)` soon.
    a1 = a[i1]
    b1 = b[i1]
    # If the components have variable length, we can't use a TensorArray anymore,
    # so use a RaggedTensor instead.
    acc1 = tf.ragged.constant([[]], dtype=a.dtype)
    def loop_test(i2, _):
      return i2 < tf.cast(a1.nested_row_lengths()[0][i1], tf.int32)
    def loop_body(i2, acc1):
      a2 = a1[i2]
      b2 = b1[i2]
      for _ in range(len(b2)):
        acc1 = tf.concat([acc1, tf.expand_dims(a2, 0)], axis=0)
      return i2 + 1, acc1

    _, acc1 = tf.while_loop(loop_test, loop_body, [0, acc1],shape_invariants=[None, tf.TensorShape([None, None])])
    acc1 = acc1[1:]  # Drop the sentinel.
    acc = tf.concat([acc, tf.expand_dims(acc1, 0)], axis=0)    # Add the row to the final result.

  acc = acc[1:]  # Drop the sentinel.
  return acc

x = tf.ragged.constant([[[b'a1', b'a2', b'a3'], [b'b1', b'b2', b'b3'], [b'c1', b'c2','c3']], [[b'd1'], [b'e1', b'e2']]])
y = tf.ragged.constant([[[b't1', b't2', b't3'], [b'u1', b'u2'], [b'v1', b'v2']], [[b'w1'], [b'x1', b'x2', b'x3']]])
print(x)
print(y)
print(tile_nd_ragged2(x, y))

I get below output
input
a = <tf.RaggedTensor [[[b'a1', b'a2', b'a3'], [b'b1', b'b2', b'b3'], [b'c1', b'c2', b'c3']], [[b'd1'], [b'e1', b'e2']]]>
b = <tf.RaggedTensor [[[b't1', b't2', b't3'], [b'u1', b'u2'], [b'v1', b'v2']], [[b'w1'], [b'x1', b'x2', b'x3']]]>
output = <tf.RaggedTensor [[[b'a1', b'a2', b'a3'], [b'a1', b'a2', b'a3'], [b'a1', b'a2', b'a3'], [b'b1', b'b2', b'b3'], [b'b1', b'b2', b'b3'], [b'c1', b'c2', b'c3'], [b'c1', b'c2', b'c3']], [[b'd1'], [b'e1', b'e2'], [b'e1', b'e2'], [b'e1', b'e2']]]>

But when i try to put @tf.function above the function and run exactly same way it gives below error
File "tf_3.py", line 101, in <module>
   print(tile_nd_ragged2(x, y))
 File "/<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 449, in __call__
   self._initialize(args, kwds, add_initializers_to=initializer_map)
 File "/<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 392, in _initialize
   *args, **kwds))
 File "/<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1847, in _get_concrete_function_internal_garbage_collected
   graph_function, _, _ = self._maybe_define_function(args, kwargs)
 File "/<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 2147, in _maybe_define_function
   graph_function = self._create_graph_function(args, kwargs)
 File "/<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 2038, in _create_graph_function
   capture_by_value=self._capture_by_value),
 File "/<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py", line 915, in func_graph_from_py_func
   func_outputs = python_func(*func_args, **func_kwargs)
 File "/<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 335, in wrapped_fn
   return weak_wrapped_fn().__wrapped__(*args, **kwds)
 File "/<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py", line 905, in wrapper
   raise e.ag_error_metadata.to_exception(e)
TypeError: in converted code:

   tf_3.py:88 tile_nd_ragged2  *
       _, acc1 = tf.while_loop(loop_test, loop_body, [0, acc1],shape_invariants=[None, tf.TensorShape([None, None])])
   /<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/control_flow_ops.py:2675 while_loop
       back_prop=back_prop)
   /<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/while_v2.py:87 while_loop
       list(shape_invariants), expand_composites=False)
   /<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/util/nest.py:536 map_structure
       structure[0], [func(*x) for x in entries],
   /<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/util/nest.py:536 <listcomp>
       structure[0], [func(*x) for x in entries],
   /<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/control_flow_ops.py:512 _shape_invariant_to_type_spec
       % shape)

   TypeError: Expected shape to be a TypeSpec or TensorShape, got None

Opening a feature request as per discussion in ticket - https://github.com/tensorflow/tensorflow/issues/38375
Feature :
support for "None" in shape_invariants for tf 1.15.x

Was able to reproduce the issue. Please find the gist here. Thanks!

@gowthamkpr i would like to make this change ? if you agree can you please point me to the code

Note that this is already resolved at head, in TF 2.2 and tf-nightly. The only question is whether there will be a 1.16 release to include it.

Update: since there will not be a TF 1.16 release, the only solution is to upgrade to TF 2.2 or newer.

As mentioned above, this already works in TF 2.2.  But for either version of TensorFlow, this can probably be rewritten to be simpler & more efficient.  As far as I can tell, the basic operation of "tile_nd_ragged2" is:
given ragged tensors a and b with matching shapes in the first two dimensions (i.e., a.row_lengths==b.row_lengths), repeat each a[i][j] value len(b[i][j]) times.
Here's an implementation of that without any loops:
def tile_nd_ragged2(a, b):
  indices = b.with_flat_values(b.values.value_rowids())
  repeated = tf.gather(a.values, indices)
  flattened_row_splits = tf.gather(repeated.values.row_splits, repeated.row_splits)
  return tf.RaggedTensor.from_row_splits(repeated.values.values, flattened_row_splits)

A few other notes, in passing:

len(a.nested_row_lengths()[0]) can be simplified to a.nrows()
In your original code, you can skip the sentinel if you use acc = tf.ragged.constant([], ragged_rank=2, dtype=a.dtype).
In tf-nightly, you can use tf.map_fn with RaggedTensors.

@edloper thank you


Cucumber integration still runs even when VELOCITY=='0'

When I set the environment variable VELOCITY to 0 or '0'
I get the TypeError: Object #<Object> has no method 'getTestsPath' from the xolvio_cumber package.
Although, I think that it may be better to have meteor-velocity control the plugins via a standard interface, the work around for removing this specific error is to have meteor-cucumber check for the environment variable. I did a quick check and it does remove the error.
Unfortunately, other parts of the testing framework do continue to function, i.e. the velocityOverlay still shows and other testing plugins, specifically mike:mocha, run.
I would be happy to contribute a PR, for the quick fix, but if the standard interface approach is preferred it seems like a more controlled roll out would be in order.

Thanks for reporting.
A PR would be very welcome. I'm a little slammed these days catching up from the holidays!


+16 endpoints on stable coverage-over-time graph

During the last conformance meeting we confirmed 16 endpoints that are not eligible for conformance testing.

https://github.com/cncf/apisnoop/blob/master/tickets/k8s/EndpointsToNotTest.org

The graph can move the existing stable/conformance line +16 points, and the related percentages.
The existing line can be made a slightly different color, does not need the calculations.
The prologue above can note this change.
/assign @zachmandeville

A PR has been opened, https://github.com/cncf/apisnoop/pull/337, that brings in the work necessary to complete this ticket.
You can find a document of this work here:
https://github.com/cncf/apisnoop/blob/tix_335/tickets/apisnoop/335_plus_16_endpoints_cot_graph.org

issue completed with merging of #337


Inherit Ownable directly from GoldToken and removed Freezable from GoldToken.

Description
GoldToken.sol should inherit directly from Ownable
Expected Behavior
Should have Ownable directly
Current Behavior
Has Ownable via Freezable
Other changes
NA
Tested
Tested by running build command locally and tests have passed as part of pipeline.
Related issues

Fixes #6662

Backwards compatibility
NA

looks like you need to import UsingRegistry.sol as well

@yorhodes Also, GoldToken is dependent on UsingRegistry by using setRegistry method. Looks things are getting complicated. What is main purpose for inheriting Ownable directly?

What is main purpose for inheriting Ownable directly?
We want the inheritance to be explicit. We should inherit from both Ownable and UsingRegistry

@yorhodes Need your input on failed test protocol-test-release which is 'Required' to be passed. Please let me if any failed test is related to this change. Thank you.

@yorhodes Need your input on failed test protocol-test-release which is 'Required' to be passed. Please let me if any failed test is related to this change. Thank you.

The automatic contract version checker failed since GoldToken has been changed without changing its version number (in its getVersionNumber function). If you're interested, you can read more about Celo's smart contract versioning here.
Just setting the version to (1, 1, 1, 1) should fix the current issue in CI.

@m-chrzan Thank you. I will check the documentation.

Thank you @nambrot for the merge that fixed the tests.

@deepakhb2  - Our Circle CI certora-test is having problems running when PRs from forked repos.  I created a replicated PR that originates from our celo-monorepo, so that certora-test will run.
Thanks for making these changes!  We will figure out how we can get that test running for PRs like yours.

Thank you @kevjue. I will use the this repo for my next issue until this issue is fixed.

I think only core developers can push to this repo. I will continue working on forked repo.


Declare this package abandonned on packagist

Hi,
this is a good practice to do it, if you are realy abandon this package, can you declare it as "abandonned" on packagist ?
Mickaël

:+1: ping @fitn @jmleroux

Done!


Thanos Ruler fails to evaluate all recording rules correctly

I am currently running Thanos v0.24.0-rc.0.
Few recording rules are evaluated fine while few recording rules seems to be last evaluated 2 days back. This happens very frequently.  Restarting the pod fixes the issue temporarily. This issue is reproducible on v0.23.0 as well.
Here is the args passed to the ruler:
- rule
- --log.level=info
- --log.format=logfmt
- --grpc-address=<IP_ADDRESS>:10901
- --http-address=<IP_ADDRESS>:10902
- --objstore.config=$(OBJSTORE_CONFIG)
- --data-dir=/thanos/data
- --eval-interval=1m
- --label=rule_replica="$(NAME)"
- --alert.label-drop=rule_replica
- --remote-write.config-file=/etc/thanos/conf/rw-config.yaml
- --query=dnssrv+_http._tcp.observatorium-thanos-query.monitoring.svc.cluster.local
- --rule-file=/etc/thanos/rules/*/*.yaml

Hello, could you please dump the goroutine stacks when this happens and upload them?

Thanks for reporting! pprof profiles available at /debug/pprof/goroutines done at the moment of things being stuck, would be super helpful!

Isn't this issue similar to https://github.com/thanos-io/thanos/issues/4772 ?

I think the .pprof file I uploaded there is actually more relevant in this issue as the behaviour I got in my Thanos Ruler is more comparable to this issue as restarting those pods helped only for some dozens of minutes before failing again to process records.
Link to the .pprof: https://github.com/thanos-io/thanos/issues/4772#issuecomment-984635383

I have the same issue on v0.24.0 (but was valid in 0.23 also)
Here the goroutine pprof after long time being stuck:
ruler-goroutine.zip


Collapse Output Data

When trying to access nested variables inside the output dictionary CloudFormation cannot directly access nested variables, i.e. trying to access the !GetAtt Resource.Address.Street with the following will fail.
{
   "Address" {
        "Street" : "Apple Street"
  }
}

So, we need to parse the response, and replace children with . notation, e.g.:
{
   "Address.Street" : "Apple Street"
}

Introduced this functionality into issue-4 branch, which has been merged into main. Unit testing works, but still requires functional testing.


Sync CARTO table with occurrence data

Created a tag carto_occurrence, so table can sync with a defined point in the version history: https://github.com/trias-project/alien-macroinvertebrates/blob/carto_occurrence/data/processed/dwc_occurrence/occurrence.csv

https://github.com/trias-project/alien-macroinvertebrates now uses new table. 9a3d47ebe5d4cebed12366bc582cde9010f8345f


Lastest updates from cosmetic packs are detacted from character in FP POV

Hello,
I thought this issue was from MoreCompany so I wrote the dev a bug. You can see screenshots of the issue here: https://github.com/notnotnotswipez/MoreCompany/issues/112

Looking into it, all cometic packs are affected
Disabling ModelReplacementAPI did solve the issue

I wanted to share the bug your way since there is a direct relationship with the issue. Thank you!

I saw there was an update 9 days ago. I ran the game with the most recent update but the issue still persists with the mod. Blow is a screenshot of cosmetics from two unrelated packs
Cosmetics Enabled:

MRA Enabled:

MRA Disabled:

Grabbed the commit from the Thunderstore site and replaced the contents of the MRA folder to check. All looks good! Thank you for the fix, @1A3Dev ! (o゜▽゜)o☆
Snaps for verification:
Cosmeics wearing

POV


BlobServiceClient::UndeleteBlobContainer doesn't support restoring a deleted container under a different name anymore.

fixed https://github.com/Azure/azure-sdk-for-cpp/issues/2252
Hi @JeffreyRichter , storage service team decided to take back the feature of restoring a deleted container under a different name. This introduces a breaking change in client side.

/azp run cpp - storage

What impact does it have on the client/SDK and to the track2 end customers? What about the previous track1 users?

What impact does it have on the client/SDK and to the track2 end customers? What about the previous track1 users?

If Track2 customers were using this feature to restore a deleted container under a different name, it doesn't work anymore.
Track1 CPP sdk doesn't support soft delete or restore container at all. So no impact to them.


Fix character loss in detail pairs with text wrapping enabled

Description
The goal of this PR is to fix https://github.com/LedgerHQ/ledger-secure-sdk/issues/749 by using the "wrapping" field of Details Context instead of forcing it to false by mistake, for calculation.
Changes include

[x] Bugfix (non-breaking change that solves an issue)
[ ] New feature (non-breaking change that adds functionality)
[ ] Breaking change (change that is not backwards-compatible and/or changes current functionality)
[ ] Tests
[ ] Documentation
[ ] Other (for changes that might not fit in any category)

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 59.88%. Comparing base (a041acd) to head (3e1eb08).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #758   +/-   ##
=======================================
  Coverage   59.88%   59.88%
=======================================
  Files          12       12
  Lines        1710     1710
=======================================
  Hits         1024     1024
  Misses        686      686

Flag
Coverage Δ

unittests
59.88% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


⚠ Bofrost.lu has degraded performance

In 934e20e, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 1899 ms

Resolved: Bofrost.lu performance has improved in 95e42db after 30 minutes.


Go SDK changed method name on 0.30.0 release

Docs Build status updates of commit 53fd188:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

tutorials/includes/go/06-get-inbox-messages.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


enum is encoded as Option, but cannot be decoded

Hi, I have the following enum:
pub enum Frame {
    Stream(Stream),
    Done,
}

When I encode Frame::Done, I get the byte 0x01. This seems correct so far, as the stream variant would be encoded as 0x00 followed by the stream (if I read the spec correctly).
However when trying to decode this message, it tries to decode as Option. And I get the error "UnexpectedEnd { additional: 1 }". Is this a limitation of bincode?
I fixed this by swapping around the Done and Stream variants in the enum definition but I'm not happy about this solution

This was actually an error on my part, my bad


Having issues installing chatterbot in python 3.8

A wonderful day to you,
I have a project to build a chat bot, but while trying to install chatterbot in my python3.8, these errors keeping stopping the process.
Installing build dependencies ... error
ERROR: Command errored out with exit status 1:
command: 'c:\users\user\pycharmprojects\mychatbot\venv\scripts\python.exe'
'c:\users\user\pycharmprojects\mychatbot\venv\lib\site-packages\pip' install --ignore-i
nstalled --no-user --prefix 'C:\Users\User\AppData\Local\Temp\pip-build-env-ic157z8y\overlay' --no-warn-script-location --no-binary :none: --only-binary :none: -i ht
tps://pypi.org/simple -- setuptools 'wheel>0.32.0,<0.33.0' Cython 'cymem>=2.0.2,<2.1.0' 'preshed>=2.0.1,<2.1.0' 'murmurhash>=0.28.0,<1.1.0' 'thinc>=7.0.8,<7.1.0'
cwd: None
Complete output (74 lines):
Collecting setuptools
Using cached setuptools-47.3.1-py3-none-any.whl (582 kB)...
Please what is the way out?

Sorry, I haven't had a chance to add Python 3.8 supported yet.

Okay, much thanks.
Please is there any suggestions for a way out or alternatives for me to meet up with deadline for my project submission?

This should be resolved now that ChatterBot version 1.0.7 was released.


[Playground] Audio player shows error on Safari

Receiving error for final stitches version of audio in audio2audio. On iPhone. Prompted with a stereo mp3.

You're on Safari right @Navenamseeg ? Can you try on Chrome?

Closing as duplicate of #77


Appearance: App Theme

Describe the bug
Files application startup, the app theme always revert back to "Windows Default" instead of the "Black" theme I have set up

Fixed in version 1.2.3.


i'm confused model architecture

https://github.com/clovaai/CRAFT-pytorch/blob/7afb2bdb62f7715662bc2457cb42d32ce5d18e2c/craft.py#L58-L80
Last stage's output shape is (H/32,W/32)
F.interpolate is look like up-sampling function.
Your model have three up-sampling function.
But predict shape is (H/2,W/2).
how can do this ?
So i have a question about this
https://github.com/clovaai/CRAFT-pytorch/blob/7afb2bdb62f7715662bc2457cb42d32ce5d18e2c/basenet/vgg16_bn.py#L42-L45
self.slice5 - Does layer have down-sampling ?

@spydr1 Thank you for error correction!
We did not realize that the size of the feature layers has been miscalculated.
Actually the conv stage2 is not the output of the max pooling but the last layer of the convolutions 2_2 in vgg16 in the following code.
https://github.com/clovaai/CRAFT-pytorch/blob/7afb2bdb62f7715662bc2457cb42d32ce5d18e2c/basenet/vgg16_bn.py#L32-L39
0 Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
1 BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
2 ReLU(inplace)
3 Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
4 BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
5 ReLU(inplace)
6 MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
7 Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
8 BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
9 ReLU(inplace)
10 Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
11 BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
12 ReLU(inplace)
13 MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
14 Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
15 BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
16 ReLU(inplace)
17 Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
18 BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
19 ReLU(inplace)
20 Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
21 BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
22 ReLU(inplace)
23 MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
24 Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
25 BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
26 ReLU(inplace)
27 Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
28 BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
29 ReLU(inplace)
30 Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
31 BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
32 ReLU(inplace)
33 MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
34 Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
35 BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
36 ReLU(inplace)
37 Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
38 BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)

Therefore, the size should be like below.

Thank you for your correction again, and sorry for the confusion.

@YoungminBaek Hello author, I found there is something wrong in your comment above:
conv stage2 is not the output of the max pooling but the last layer of the convolutions 2_2,yeah, this is right !
but, according to your print info above:
conv stage3 is the output of the conv3_2,  not conv3_3;
conv stage4 is the output of the conv4_2,  not conv4_3;
conv stage5 is the output of the conv5_2,  not conv5_3;
is it your miscalculation or my misunderstanding?

in addition, could you tell me the value of sigma in gaussian kernel?


Why a proprietary attachments API?

Why doesn't the gem use the ActionMailer attachments API, or at least capture it if the hash is present? It seems like the parameters involved are identical or a subset:
http://api.rubyonrails.org/classes/ActionMailer/Base.html#method-i-attachments

Really why?

@iSabanin @wildbit any thoughts on this?


1.0.3: cannot pass buffer/string certificates to https option anymore

Operating System: macOS 10.13.3
Node Version: 8.11.1
NPM Version: 5.6.0
webpack Version: 4.11.1
webpack-serve Version: 1.0.3

This issue is for a:

[x] bug
[ ] feature request
[ ] modification request

Code
webpack.config.js
import startDevServer from 'webpack-serve';
import { certificateFor } from 'devcert';

// https holds buffers
const https = await certificateFor('localhost', {
 skipHostsFile: true,
 skipCertuil: false,
});

startDevServer({
 // other config does not matter, reproducible either way
 https,
});

Expected Behavior
Should start up the dev-server with a valid https cert.
Actual Behavior
Throws an error:
An error was thrown while starting webpack-serve.    { Error: ENAMETOOLONG: name too long, lstat 'Certificate: {...certificate data}
How Do We Reproduce?
Generate a self-signed certificate using devcert and pass it in as https option to webpack-serve >= 1.0.3.
Before 1.0.3 it was perfectly fine to pass buffers/strings for each entry in the https options, but since 1.0.3 it does not work anymore.
I think this bug was introduced in https://github.com/webpack-contrib/webpack-serve/commit/e030696cf0c64c97582f33c1ae3492918c23fb41#diff-763aa8deb604d725a74f8304903c1422R78
It basically runs fs.lstatSync() on the actual passed string/buffer value of the cert/key, which of course is too long. We'll need to check first weather it's an actual file (path) before using lstat.

Are you sure you're using/reporting the right project? This project hasn't gotten to version 4 yet.

@shellscape

Are you sure you're using/reporting the right project? This project hasn't gotten to version 4 yet.

😄 My bad. Of course I meant 1.0.3 instead of 4.0.3.
Edited.
Thanks for pointing that out!

@stnwk ah, there we go. would you be interested in submitting a fix for this? should be a simple test of whether or not the value is a Buffer.

Sure, i'll take a look at it! :)

@shellscape Was about to submit my PR when I realised you fixed it yourself.. what's the point of asking for help then? 😕 Heads-up would have been nice.
Either way: Thanks for tackling this so quickly!

@stnwk we welcome contribution whole heartedly, but we also can't let critical bugs like that stand unresolved for too long. you might also run into the same situation where another user submits a PR for a fix at the same time you're working on one. unfortunately without opening a pre-emptive work-in-progress PR, there's no way to know who's working on what or how close to ready a fix is. sorry that this one got fixed before you had that chance.


Support non-uniform categoricals

This adds support for non-uniform categoricals (different categories in each partition) to dask.dataframe. Currently this is written to only support pandas 0.19.0. It could be made backwards compatible, but we may also want to discuss dropping support for < 0.19.0.
Fixes #1836.
TODO:

[x] Support non-uniform categoricals in partitions
[x] Basic functionality works (reductions, groupby, etc...)
[x] categorical accessor
[x] get_dummies
[x] pivot_table
[x] concat
[x] append
[x] merge
[x] read_csv (as per https://github.com/pandas-dev/pandas/pull/13406)
[x] hash_pandas_object is categorical-mapping agnostic
[x] Fix bug in partd
[x] Maybe make backwards compatible with pandas 0.18?

This looks cool.  It would be nice to remove this roadblock when using categoricals with dask.dataframe.
cc @TomAugspurger @sinhrks

What is the status here?  @jcrist are you waiting on feedback?  Please let me know if this is blocking on me.

No, I'm still working on this.

Ok, this should be ready for review. A quick walk through of what's added here:
This adds support for different partitions having different categorical mappings. This is accomplished through union_categoricals, a method added in pandas 0.19.0. As such, support for pandas < 0.19.0 has been dropped.
Changes to metadata:
Categorical columns/indices now have two states:

known categoricals have the categories known statically (on the _meta attribute)
unknown categoricals don't know the categories statically, and are indicated by the presence of dd.utils.UNKNOWN_CATEGORIES in the categories on the meta attribute. This is nice, because operations on the categorical in most cases propagate the presence of this value, so the known/unknown state carries through inference.

Changes to the categorical accessor
To query and convert between these states, following has been added to the categorical accessor:

.cat.known is a property that indicates if the categories are known
.cat.as_known() converts unknown to known (and is a no-op if already known). Note that this requires computation.
.cat.as_unknown() converts known to unknown. This requires no computation - it's just a change in metadata.

If the categories are known, then .cat.categories returns statically (no computation needed). If they are unknown, then an error is raised (rather than computing them)
Changes to concat
Concatenation of dask dataframes with different categoricals is now supported. For each column/index in a dataframe, the output dask column/index will have known categories only if all categories are known. Otherwise the categories will be unknown.
After a call to concat, each partition is guaranteed to match the metadata on the containing dask dataframe. This fixes a bug from before where in certain cases partitions could have different columns/datatypes.
Changes to pivot_table and get_dummies
These functions only work if the categories are known. If unknown, an error is raised
Changes to read_csv
For pandas > 0.19.2, dask now supports reading from csvs directly into categorical columns. These will be unknown categories
Changes to merge
No changes are made to merge. Pandas merge doesn't support merging categoricals, even if the categoricals are identical. As such, dask doesn't error, but the categorical datatype is dropped. When this is fixed in pandas, support for merge on differing categoricals can be added.
Changes to hash_pandas_object
Previously, this would hash categories based on codes only. We now hash based on the values the categoricals represent. This was patched in pandas, and will be in the next release. The fix is backported to dask for versions 0.19.0 to 0.19.2. The faster (and more stable) object hash will be used for pandas 0.19.2.

And a quick demo using the airline dataset:
In [1]: import pandas as pd

In [2]: dtypes = pd.read_csv('1987.csv', dtype={'Origin': 'category', 'Dest': 'category'}).dtypes.to_dict()

In [3]: import dask.dataframe as dd

# The dtypes to `Origin` and `Dest` are `category`, so they're
# read directly in as categoricals
In [4]: ddf = dd.read_csv('198*.csv', dtype=dtypes)

In [5]: ddf
Out[5]: dd.DataFrame<from-de..., npartitions=18>

In [6]: ddf.Origin.dtype
Out[6]: category

In [7]: ddf.Origin.cat.known    # categories are unknown at this point
Out[7]: False

In [8]: worst_avg_delay = ddf.DepDelay.groupby(ddf.Origin).mean().nlargest(10)

In [9]: worst_avg_delay.index.dtype
Out[9]: category

In [10]: worst_avg_delay.index.cat.known
Out[10]: False

In [11]: worst_avg_delay.compute()
Out[11]:
Origin
PIR    85.000000
ABI    36.500000
EGE    32.363636
BFI    28.000000
ROP    24.700000
ACV    19.740223
SUN    19.000000
YAP    16.902482
RDD    15.724234
HDN    14.147350
Name: DepDelay, dtype: float64

# Convert unknown categoricals to known
In [12]: known_origin = ddf.Origin.cat.as_known()

In [13]: known_origin.cat.known
Out[13]: True

In [14]: known_origin.cat.categories
Out[14]:
Index(['ABE', 'ABQ', 'ACV', 'AGS', 'ALB', 'ALO', 'AMA', 'ANC', 'APF', 'ATL',
       ...
       'PIR', 'ACY', 'GST', 'DET', 'BFI', 'ROP', 'ABI', 'TVC', 'EGE', 'SUN'],
      dtype='object', length=246)

Nice!  I would prefer .cat.known to be .cat.is_known since it's boolean, which would follow how pandas does this with other boolean attributes.

I would prefer .cat.known to be .cat.is_known since it's boolean, which would follow how pandas does this with other boolean attributes.

The categorical accessor .cat.known property and .cat.as_known/.cat.as_unknown methods mirror the same for ordered (.cat.ordered, .cat.as_ordered, .cat.as_unordered). I think the consistency here is good, but happy to change if others feel strongly.

Oh, good comparison.  Thanks.  .cat.known is fine with me.

Open questions:

Currently only Series/Index objects have a way to convert unknown categoricals to known (and vice-versa). One option would be to hijack the existing df.categorize method to convert unknown to known. I'm not sure what the reverse would be (or if it'd be needed).

Currently only Series/Index objects have a way to query the known/unknown status of categoricals. I'm not sure if this is needed on a dataframe. If it is, I think the following interface might be good:
is_categorical(df, known=None)   # default, returns true for all categoricals
is_categorical(df, known=True)    # returns true only for known categoricals
is_categorical(df, known=False)   # returns true only for unknown categoricals

If needed, I'm not sure if it should be a method, or a function in utils.

In general, I think I'd rather be conservative with the public api, and add things as they're needed.

Ping @mrocklin, @sinhrks, and @TomAugspurger for review.

Can I ask for a small section in the dataframe docs about categoricals?  Your explanation here about known and unknown is very useful.  I expect that we'll get questions about this downstream.  It would be useful to have a link to which we could point people.

Sure. I think I'll wait until the code is reviewed (and any changes made) before adding the docs.

Thanks for the review. Both of those issues are caused by x being a categorical of type categorical (as seen in the repr of output 125). This is due to categorize recategorizing an already categorized categorical. Will fix.

I don't know how big a deal this is, but in this case Index.as_known fails
In [217]: s = dd.from_pandas(pd.Series(pd.Categorical(['a', 'b', 'a', 'b']), name='name'), npartitions=2).cat.as_unknown()

In [218]: vc = s.value_counts()

In [219]: vc.index.cat.as_known()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/Users/tom.augspurger/Envs/py3/lib/python3.6/site-packages/dask/dask/dataframe/utils.py in raise_on_meta_error(funcname)
    147     try:
--> 148         yield
    149     except Exception as e:

/Users/tom.augspurger/Envs/py3/lib/python3.6/site-packages/dask/dask/dataframe/core.py in _emulate(func, *args, **kwargs)
   2928     with raise_on_meta_error(funcname(func)):
-> 2929         return func(*_extract_meta(args, True), **_extract_meta(kwargs, True))
   2930

/Users/tom.augspurger/Envs/py3/lib/python3.6/site-packages/dask/dask/compatibility.py in apply(func, args, kwargs)
     46         if kwargs:
---> 47             return func(*args, **kwargs)
     48         else:

/Users/tom.augspurger/Envs/py3/lib/python3.6/site-packages/dask/dask/dataframe/core.py in _reduction_chunk(x, aca_chunk, **kwargs)
   3557 def _reduction_chunk(x, aca_chunk=None, **kwargs):
-> 3558     o = aca_chunk(x, **kwargs)
   3559     # Return a dataframe so that the concatenated version is also a dataframe

/Users/tom.augspurger/Envs/py3/lib/python3.6/site-packages/dask/dask/dataframe/methods.py in cat_categories(x)
    141 def cat_categories(x):
--> 142     return x.cat.categories
    143

AttributeError: 'CategoricalIndex' object has no attribute 'cat'

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-219-2b7b72789188> in <module>()
----> 1 vc.index.cat.as_known()

/Users/tom.augspurger/Envs/py3/lib/python3.6/site-packages/dask/dask/dataframe/accessor.py in as_known(self, **kwargs)
    175         if self.known:
    176             return self
--> 177         categories = self._series.reduction(methods.cat_categories, M.unique)
    178         categories = compute(categories, **kwargs)[0]
    179         return self.set_categories(categories)

/Users/tom.augspurger/Envs/py3/lib/python3.6/site-packages/dask/dask/dataframe/core.py in reduction(self, chunk, aggregate, combine, meta, token, split_every, chunk_kwargs, aggregate_kwargs, combine_kwargs, **kwargs)
    681                    split_every=split_every, chunk_kwargs=chunk_kwargs,
    682                    aggregate_kwargs=aggregate_kwargs,
--> 683                    combine_kwargs=combine_kwargs, **kwargs)
    684
    685     @derived_from(pd.DataFrame)

/Users/tom.augspurger/Envs/py3/lib/python3.6/site-packages/dask/dask/dataframe/core.py in apply_concat_apply(args, chunk, aggregate, combine, meta, token, chunk_kwargs, aggregate_kwargs, combine_kwargs, split_every, split_out, split_out_setup, split_out_setup_kwargs, **kwargs)
   2885
   2886     if meta is no_default:
-> 2887         meta_chunk = _emulate(apply, chunk, args, chunk_kwargs)
   2888         meta = _emulate(apply, aggregate, [_concat([meta_chunk])],
   2889                         aggregate_kwargs)

/Users/tom.augspurger/Envs/py3/lib/python3.6/site-packages/dask/dask/dataframe/core.py in _emulate(func, *args, **kwargs)
   2927     """
   2928     with raise_on_meta_error(funcname(func)):
-> 2929         return func(*_extract_meta(args, True), **_extract_meta(kwargs, True))
   2930
   2931

/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/contextlib.py in __exit__(self, type, value, traceback)
     98                 value = type()
     99             try:
--> 100                 self.gen.throw(type, value, traceback)
    101                 raise RuntimeError("generator didn't stop after throw()")
    102             except StopIteration as exc:

/Users/tom.augspurger/Envs/py3/lib/python3.6/site-packages/dask/dask/dataframe/utils.py in raise_on_meta_error(funcname)
    159                ).format(" in `{0}`".format(funcname) if funcname else "",
    160                         repr(e), tb)
--> 161         raise ValueError(msg)
    162
    163

ValueError: Metadata inference failed in `apply`.

Original error is below:
------------------------
AttributeError("'CategoricalIndex' object has no attribute 'cat'",)

Traceback:
---------
  File "/Users/tom.augspurger/Envs/py3/lib/python3.6/site-packages/dask/dask/dataframe/utils.py", line 148, in raise_on_meta_error
    yield
  File "/Users/tom.augspurger/Envs/py3/lib/python3.6/site-packages/dask/dask/dataframe/core.py", line 2929, in _emulate
    return func(*_extract_meta(args, True), **_extract_meta(kwargs, True))
  File "/Users/tom.augspurger/Envs/py3/lib/python3.6/site-packages/dask/dask/compatibility.py", line 47, in apply
    return func(*args, **kwargs)
  File "/Users/tom.augspurger/Envs/py3/lib/python3.6/site-packages/dask/dask/dataframe/core.py", line 3558, in _reduction_chunk
    o = aca_chunk(x, **kwargs)
  File "/Users/tom.augspurger/Envs/py3/lib/python3.6/site-packages/dask/dask/dataframe/methods.py", line 142, in cat_categories
    return x.cat.categories

I think converting just the index to known will be a less common / useful operation.

Ooh, that has to do with dask having Index as a subclass of Series, while pandas doesn't. I bet none of the categorical accessor methods work on categorical indices. That's fixable too.

Yeah, basically comes down to pandas CategoricalIndex using .categories, .codes, etc. directly instead of under the .cat accessor right? That's a bit unfortunate.

Yes. I'm not sure if we should mimic that in the case of categorical index type, or go with the categorical accessor. Perhaps both.
In general, there are several methods that dask indices support that pandas indices don't, since the dask Index class subclasses from Series.

One question (would be nice to confirm in the docs): .cat.codes.compute() on a categorical should be "correct" in that all the category: code mapping should be consistent. It looks like that's the case here, but want to make sure that's not a fluke:
In [300]: s = dd.from_pandas(pd.Series(pd.Categorical(['c', 'b', 'b', 'a']), name='name'), npartitions=2).cat.as_unknown()

In [301]: s.cat.codes.compute()
Out[301]:
0    2
1    1
2    1
3    0
dtype: int8

Ooh, that's a good point, I didn't think of that one. This is actually only for categoricals that are known. For unknown categoricals, we'd need need a unifying step, which can't be done without a full dataset traversal. My preference is to error in this case, rather than doing something inefficient.

I believe all comments have been addressed.
One new change that people may want to look at: previously we'd wrap all output from the accessors in calls to pd.Index/pd.Series if the return type was a numpy array. This caused differences between the pandas and dask interfaces. Since we now support arrays with unknown shape, I switched these to return dask arrays. This was part of a larger refactor of how the accessor class is implemented, and how the methods get forwarded to the dd.Index class.

Failure is due to groupby not supporting grouping by array. Looks like an easy fix (we already support conversion). It's late here, will fix tomorrow.

FYI, I won't have a chance to review this any more closely till next week, so I'm +1 to merge when you're ready. Great work!
On Fri, Jan 20, 2017 at 6:26 PM -0600, "Jim Crist"<EMAIL_ADDRESS>wrote:
Failure is due to groupby not supporting grouping by array. Looks like an easy fix (we already support conversion). It's late here, will fix tomorrow.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

So it turns out that having the accessor's return arrays (when they do in pandas) isn't the best thing, because we lose the division information (we actually have a groupby test with an accessor and known divisions). So I need to revert the behavior here :(.
Q: Previously, when an accessor returns a numpy array, we'd wrap it in the same type as the partition type (so for foo that returns a numpy array in numpy, index.foo -> index, series.foo -> series). All but one of these methods return a numpy array only on indices. I'm not sure if this is the best way to do this, or if we should be consistent with what we wrap numpy arrays with (e.g. always series/always index).
Right now I'm leaning towards always wrapping numpy array return values in Series. Thoughts @sinhrks, @TomAugspurger?

Ok, I reverted to the previous behavior, where x.accessor.foo returns the same type as x if pandas would return a numpy array. However, we now also set the index as equivalent in the case where x is a series. This is needed to support operations where pandas would align indices before computation (e.g. groupby).
After further though, I think this is for the best. For many operations, pandas treats arrays and indices effectively as equivalent (e.g. not aligning on index beforehand), making them a better match than series. For the one series operation that returns a numpy array (str.to_pydatetime) I figured it made sense to return a series, as it matches the other string accessor methods.

All tests passed, I think this is good to go now.

@jcrist

str.to_pydatetime

you mean .dt.to_pydatetime right? and this returns a ndarray for compat with numpy. To be honest, dask really doesn't have a need for this at all. personally I would remove it (and any non Series/Index routines)

you mean .dt.to_pydatetime right?

Oop, yes, that's what I meant.

personally I would remove it (and any non Series/Index routines from accessors).

Many of the index methods/properties that match those on the cat/dt accessor return numpy arrays for indices. Since we'd like to support things like df.index.day, then this is needed. The maintenance cost here is minimal, and I think the decision to return indices makes sense.

@jcrist sure makes sense. I think pandas will soon also simply return Index/Series for these in any event; the reason they do now is compat.

I just gave this PR a quick test drive and was quite pleased with the experience.

I'm going to merge this now, and add docs in a follow-up PR. I think it'd be good to get this on master now so others can try it out easier. Any changes/additional features can be added as needed later.

I'm glad to see this in.  Thanks taking on a large task like this @jcrist .


Added clear graph button.

Closes issue #31
I went ahead and added the feature to try to help out a bit.
Feel free to ask for changes if you feel they're needed.

Thanks for the PR! It's merged on develop and should land on master this week end as part of version 0.3

Of course! Happy to help out, especially with such a cool project :+1:


#451: Add java friendly api for PipelineFlowFactory

Add Java friendly PipelineFlowFactory API.
Add additional javadsl HttpRequest/Response APIs into RequestContext
Thanks for your pull request.  Please review the following guidelines.

[x] Title includes issue id.
[x] Description of the change added.
[x] Commits are squashed.
[x] Tests added.
[x] Documentation added/updated.
[ ] Also please review CONTRIBUTING.md.

The java tests that are added here for PipelineFlow mimic the scala specs we already had.

@agursel I have updated the PR with the delta you suggested.  Still two commits on this PR here.  If it looks good please squash and merge.


JSON Schema generator: Add support for NodaTime

[ ] Add here: https://github.com/RSuter/NJsonSchema/blob/master/src/NJsonSchema/Generation/JsonObjectTypeDescription.cs#L79

=>Dynamic (without assembly/nuget reference)
@MaxDeg
Well there a lot of types for NodaTime, not only those 2.
http://nodatime.org/2.2.x/userguide/core-types

https://github.com/RSuter/NSwag/issues/923


mesa: fix intel-clc not taking disabled

Description of changes
Fixes Mesa cross compiling for gnu64 when on aarch64 and on Asahi.
Fixes #314457

Things done

Built on platform(s)

[ ] x86_64-linux
[x] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

We don't support building arbitrary versions of packages in Nixpkgs.  This should either be worked around downstream, or you can wait for #306791.

(To work around this downstream in an overlay, you could use filterAttrs to filter out Nixpkgs' intel-clc option, and then append your own.)

This PR doesn't fix my actual problem. https://github.com/tpwrules/nixos-apple-silicon/pull/203 fixes my actual problem.


Change the /target line in .gitignore generated by cargo new to /**/target/

Describe the problem you are trying to solve

Ignore all target directories within a directory at infinite depth with cargo new. This is useful as projects grow, without manually needing to add target directories to .gitignore, or developers figuring out how to fix many files being added in a target directory with pull requests. At the moment the generated file for .gitignore with cargo new hi is as shown:
[jray@jr-dl r]$ cargo new hi
     Created binary (application) `hi` project
[jray@jr-dl r]$ cd hi
[jray@jr-dl hi]$ cat .gitignore
/target
**/*.rs.b

Describe the solution you'd like

Change the /target line above to /**/target/, like with https://github.com/github/gitignore/pull/2866.
Notes

Similar to https://github.com/rust-lang/cargo/issues/2565.

It's intentional that it doesn't end in / so that it can be a symlink, so we probably don't want to undo that.

Fair enough.

Having a submodule src/target/mod.rs doesn't seem like an entirely uncommon module name. An alternative would be to have cargo new create a sub-.gitignore inside new crates folders when run in an existing git repo (maybe even detecting if you have a workspace and only adding it if not).

Alternatively in such cases you can then modify .gitignore to then exclude such folders.

Perhaps

target
!/**/src/**/target/

to allow target to be a symlink (as discussed above).

See discussion in https://github.com/github/gitignore/pull/2866, summarised with https://github.com/github/gitignore/pull/2866#issuecomment-443056387.


Add support for Zeppelin 0.7.0

New repo/packages/Z/zeppelin/4 directory. Requires a new mesosphere/zeppelin:0.7.0, as in this PR: https://github.com/mesosphere/dcos-zeppelin/pull/15

Please bump Spark to 1.0.9-2.1.0-1-hadoop-2.6
Ref: https://hub.docker.com/r/mesosphere/spark/tags/

Also, I don't actually see the new Zeppelin Docker image tag: mesosphere/zeppelin:0.7.0 at https://hub.docker.com/r/mesosphere/zeppelin/tags/ yet?

I just pushed an update for the Spark tag.
The Zeppelin 0.7.0 image requires this PR: https://github.com/mesosphere/dcos-zeppelin/pull/15.

@deanwampler LGTM apart from the Zeppelin Docker  image tag 0.7.0.
Will merge when https://github.com/mesosphere/dcos-zeppelin/pull/15 gets merged.

Awesome. Thanks!

@deanwampler The package/package.json and package/resource.json will be base64 encoded into the DCOS_PACKAGE_METADATA for DC/OS (specifically the Marathon) to display info. The package/marathon.json.mustache determines how Marathon will run container and what environment variables to set for the container. The package/config.json is what you can set through the Marathon package setup interface, such as docker image for Zeppelin service, environment variables you want Marathon passes into container. Therefore, the pcakge/config.json and the package/marathon.json.mustache come hand in hand.
For zeppelin 0.7.0, you can add the "default": "mesosphere/spark:1.0.9-2.1.0-1-hadoop-2.7" to the package/config.json under the "executor_docker_image" section and the "default": "https://downloads.mesosphere.com/spark/assets/spark-2.1.0-bin-2.7.tgz" to the package/config.json under the "spark" -> "properties" -> "uri" section.

I just pushed an update with those changes, @jshenguru.

I forked the dcos-zeppelin project (https://github.com/jshenguru/dcos-zeppelin) and change from package-based to Marathon app based manner and made a zeppelin 0.7.0 docker image (https://hub.docker.com/r/jshenguru/zeppelin/). There is no need for wait anymore.

@deanwampler do you want to update the package to use the image that @jshenguru created?

Is there any progress here @jshenguru @deanwampler? What needs to be done to finalize this?
cc @ryadav88

I just pushed an update to use @jshenguru's docker image for Zeppelin and use the latest mesosphere Spark plugin.

I just pushed an update for both changes.


Feature request: Huggingface tokenizer support

Since world tokenizer training code is not available as far as I know, those of us who need a custom tokenizer train HF tokenizer (pip rwkv package, RWKV-LM trainer and json2binidx_tool all support it).
Currently it doesn't work with ai00_server:

[ai00_server::middleware] reload model failed: failed to parse vocabulary: invalid value: expected key to be a number in quotes at line 2 column 3

Implemented this, pull request: https://github.com/cryscan/web-rwkv/pull/23


feat: Support Moore Threads GPU

Since pull request #8383 has been merged into llama.cpp, I would like to add MTGPU support to GPUStack. This PR might be the first step towards that goal.

Thanks.
I am happy to see this. GPUStack is based on https://github.com/gpustack/llama-box, and I tried to compile a distribution for MT, see https://github.com/gpustack/llama-box/commit/0907d661bf98e6a74a446e16a8dccdffe7face80.
But it failed. If you can, please fix the llama-box. After llama-box generates the MT version correctly, I will approve this and go ahead.

Thanks.
I am happy to see this. GPUStack is based on https://github.com/gpustack/llama-box, and I tried to compile a distribution for MT, see gpustack/llama-box@0907d66.
But it failed. If you can, please fix the llama-box. After llama-box generates the MT version correctly, I will approve this and go ahead.

Sure! Here's a refined version of your GitHub comment:
That sounds reasonable. I can assist with that.

It seems that yesterday's CI failure is unrelated to this PR (fastfetch was mistakenly sourced from a fork).

@thxCode Could you approve the workflows? Thanks.

Yes, I've already started on this. As mentioned in the PR description, this is the very first step, which is why I've marked MTGPU in the future support list.

@yeahdongcn Does MT driver implement all the same userland APIs as NV's, including CUDA and NVML?

@yeahdongcn The new feature you're proposing is indeed valuable. Before we can merge the PR into GPUStack, need to ensure its dependencies (fastfetch and llama-box) are working correctly.
Once the dependencies are verified, we will promptly revisit and merge your PR.

@yeahdongcn Does MT driver implement all the same userland APIs as NV's, including CUDA and NVML?

Not exactly the same, but similar.

@yeahdongcn Does MT driver implement all the same userland APIs as NV's, including CUDA and NVML?

Not exactly the same, but similar.

Will Old Huang sue you?

See https://github.com/gpustack/fastfetch/pull/2 for fastfetch changes.

@yeahdongcn Does MT driver implement all the same userland APIs as NV's, including CUDA and NVML?

Not exactly the same, but similar.

~Will Old Huang sue you?~
Just kidding

Haha, you might want to check how HIP did it.

@yeahdongcn I think MT provides the same APIs on Windows as what it does on Linux. The fastfetch part can be ported to Windows easily, right?
@aiwantaozi The current way of GPU detection on Windows is kinda unfriendly for non-Nvidia GPUs. I have solved the multi-gpu issue in the official fastfetch repo ( still needs more testing though ). You may backport it to your fork repo.
https://github.com/fastfetch-cli/fastfetch/blob/884fd097c61933bd441ed3ea65c8d3f4797c3bf8/src/detection/gpu/gpu_windows.c#L152-L180

@yeahdongcn I think MT provides the same APIs on Windows as what it does on Linux. The fastfetch part can be ported to Windows easily, right?
@aiwantaozi The current way of GPU detection on Windows is kinda unfriendly for non-Nvidia GPUs. I have solved the multi-gpu issue in the official fastfetch repo ( still needs more testing though ). You may backport it to your fork repo.
https://github.com/fastfetch-cli/fastfetch/blob/884fd097c61933bd441ed3ea65c8d3f4797c3bf8/src/detection/gpu/gpu_windows.c#L152-L180

Yeah, but I'm afraid I'm not a Windows guy.

@CarterLi ca
Due to time constraints, we haven't had a chance to report and fix bugs upstream yet. We are very grateful to you and your project for the significant help. Since GPUStack will run as a Windows Service, we cannot obtain device information through EnumDisplayDevicesW when running as a Windows Service. Therefore, we made some temporary modifications. We would greatly appreciate any suggestions you might have. Thank you!

@yeahdongcn
Hi,
fastfetch has released <IP_ADDRESS>, could you please update and check if it works with GPUStack?
After updating the version here, could follow the development doc. Thanks.

Since GPUStack will run as a Windows Service, we cannot obtain device information through EnumDisplayDevicesW when running as a Windows Service. Therefore, we made some temporary modifications. We would greatly appreciate any suggestions you might have. Thank you!

In this case, you may enumerate PCI devices by enumerating HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\PCI or setupapi.
Note: not every GPU uses PCI bus. Notably Apple Silicon

@CarterLi Thank you for your fantastic suggestions! Will try to get GPU info from Windows registry.

Upstream now uses SetupAPI to fetch GPU devices. It should work when running as a Windows Service ( untested though ) https://github.com/fastfetch-cli/fastfetch/commit/0354fd5d9de1d70778cbd3e16778511ec4f5d372
I suggest you move back to upstream, instead of maintaining a dirty and outdated fork. If you encounter any bug you can either open PR to fix it, or just file it upstream and I will look at it.
The only one upstream is missing is UUID, which is unnecessary IIRC. We have FFGPUResult::deviceId. It's AdapterLuid on Windows and bus address on Linux, both should not change.

@yeahdongcn Hi, fastfetch has released <IP_ADDRESS>, could you please update and check if it works with GPUStack?
After updating the version here, could follow the development doc here to run. Thanks.

It looks good after switching fastfetch to my local build (with PR #3). Please see the screenshot below.

One notable issue is that our MTML library currently has no interface to query the number of cores. I have opened a JIRA ticket internally and will make another change to fix this.


Fix signed commits CI job

The job was previously ran on "pull_request_target". This runs the job on the base branch (i.e. the main branch of the
1password/shell-plugins repo). Since we want to know if commits are signed on all the commits to be merged (i.e. also on the fork), I have changed this job to run on the merge commit instead.
Relevant documentation
https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request_target
Note that the usage example of https://github.com/1Password/check-signed-commits-action also recommends running on the "pull_reqeusts" event.
Use Cases
See https://github.com/1Password/shell-plugins/pull/332 where we run into this in practice. The job is passed, but commits aren't signed.

The change from pull_request to pull_request_target was needed to support PR commenting on PRs from forks. Otherwise the GitHub token mounted in the fork pull_request workflow does not have permissions to comment in the upstream repo.
However, that change alone didn't fix the commenting, because pull_request_target breaks the PR number check, which results in the No PR found to scan for commits. you're seeing.
There's a PR open in the action repo that in turn fixes that. I'll make sure to also update the examples/docs in the action repo.

However, that change alone didn't fix the commenting, because pull_request_target breaks the PR number check, which results in the No PR found to scan for commits. log message you're seeing.
There's a https://github.com/1Password/check-signed-commits-action/pull/4 in the action repo that in turn fixes that. I'll make sure to also update the examples/docs in the action repo.

Looks like this didn't work. The job still exists with the same log message.

Looks like this didn't work. The job still exists with the same log message.

That's because it's using pull_request_target which is main and thus not this branch. So in practice, any changes made to this particular workflow will be ignored until they're merged to main.


[PROPOSAL] Support for Static lineage in the spec

Purpose:
Today OpenLineage captures metadata mainly in the context of a run that is actively accessing and modifying/producing data. The metadata collected covers both dynamic aspects that are specific to the run (how long it took, how many rows were produced, ...) and static metadata that does not change every time the job runs (inputs, outputs, dataset schema, code version, ...).
We're referring to those two types of metadata as dynamic and static.
We want to collect that static metadata for Datasets and Jobs outside of the context of a run through OpenLineage.
Examples:

defining dataset owners
marking a dataset as containing personally identifying information.

Proposed implementation
The proposed implementation is to add two new types of events to the spec:

Job event: contains input datasets, output datasets, and job static facets.
Dataset events: contains static metadata dataset facets.

run.facets, dataset.inputFacets and  dataset.outputFacets are dynamic metadata about the specific run.
job.facets, dataset.facets are static metadata.

Submitted: https://github.com/OpenLineage/OpenLineage/pull/1839

I would suggest adding another timestamp field to static facets to allow for soft deletes.
Aren't some facets static by definition, do they need a new field to identify them? I would think this is the responsibility of the open lineage consumer implementation to process facets, rather than the job of the emitter to correctly identify what should be static.
What if emitters just emitted an 'OTHER' eventType for additional metadata tagging? The ol implementation can disregard it as part of the lineage DAG, since it has no START event. So the view would be that ol metadata maintainers are just another ol emitter. If they are only modifying datasets, their jobid could be the maintainers unique id as an origination identifier.


Decimal number grouping

#38 has used twitter_cldr to implement number localisation, with thousands grouping, but twitter_cldr does not implement groupings of numbers in decimals, which BIPM in their wisdom also need.
For that reason, unless someone else out there has done it, we need to implement a number formatter to replace twitter_cldr.
This number formatter will take the following configuration options, exemplified from the twitter_cldr YAML configuration files:
    :decimal: "." — choice of decimal point
    :exponential: E — for exponential notation
    :group: "," — for groupings of three integer digits
    :fraction_group: "," — for groupings of three fractional digits—not in twitter_cldr
    :minus_sign: "-" — for exponential notation, negatives
    :nan: NaN — for floating point notation
    :plus_sign: + — for exponential notation

The formatter should deal with decimals and integers for now; we can implement exponential notation later.
We also should not hardcode the number of integer and fractional digits to group; Indian scripts use groups of two instead of three.
The formatter should take either a numeric or a string argument. If string, we can assume English locale for float (i.e. decimal point not comma). If string, the number of decimal points should be the same as what is passed in; if numeric, there should be a precision option for how many digits to print, defaulting to a value like 3.
The number formatter will be a new gem under plurimath.
So:
NumberFormatter(1000.01212, decimal: ",", group: " ", fraction_group: " ", group_digits: 2, fraction_group_digits: 2, precision: 3) = "10 00,01 21"

NumberFormatter("1000.01212", decimal: ",", group: " ", fraction_group: " ", group_digits: 2, fraction_group_digits: 2) = "10 00,01 21 2"

And for now the option hashes to be used by the formatter are: decimal, group, fraction_group, group_digits, fraction_group_digits, precision.

As mentioned:
It is possible that you can set default values for the options hash using twitter cldr and locale
require "twitter_cldr"
locale =<EMAIL_ADDRESS>? @lang.to_sym : :en
num = TwitterCldr::DataReaders::NumberDataReader.new(locale)

num.symbols includes some of the entries in the hash. So you could use locale as an argument to look up default values, but any explicit values in the hash would still override them.
So for :en, :decimal == ".", for :fr, :decimal == "."

So for :en, :decimal == ".", for :fr, :decimal == "."

You mean :en, :decimal == ".", for :fr, :decimal == ","?

Yeah, that.

@opoudjis implemented almost all requirements except one: group_digits, this value is usually set by cldr notation for particular locale, so I don't think its will be normal to change this behavior. Take a look: metanorma/isodoc#223

group_digits is not essential, it was only there in case we did reimplement CLDR.

... Wow, great work!


Merge 5.0 breaking changes into the main line of development

This PR merges the 5.x branch so that it becomes the main line of development in preparation for the 5.0.0 release.

Reverted the change that was breaking the tests.


MentionsInput don't support props maxLenght

why MentionsInput don't support maxLenght , after all it works just like simple teaxtarea/input

@alenka1710 just to make sure, since you misspelled length two times in this issue, did you spell is maxLength instead of maxLenght? We pass all props through, that we don't consume, so maxLength should be supported.