Suggestion: Void-protected types

Deep in the discussion of non-nullable types, there was a new, more practical, suggestion (credit particularly to @tejacques) of voidable types that are protected against unsafe access, while maintaining backwards compatibility. I think the suggestion is a bit too buried in that thread to be useful or noticed, so I'm creating a new issue for this proposal.
First, some definitions to make sure I'm on the same page as everyone else... for the purposes of this discussion, "void" refers to either a value of null or undefined, which will raise a TypeError if you attempt to use it in certain ways, particularly accessing a member (e.g. foo.bar) or calling it as a function (e.g. foo()). Currently in TypeScript, all types are "voidable", by this definition.
The proposal is for a new modifier (I'll use a ? prefix, similar to Flow's implementation, but it could be anything) that will make a type "void-protected". A void-protected type must be checked for existence before it can be used. For example:
function printShout(x: ?string): void {
  console.log(x.toUpperCase()); // Error: x might be void
  if (x) {
    console.log(x.toUpperCase()); // legal
  }
}

This does not affect assignments, for the sake of backwards compatibility:
function foo(x: ?string): void {
  var y: string = x;
  console.log(y.toUpperCase()); // Legal, but probably not a good idea
}

This is similar in philosophy to Java's Optional<> type. While every type in Java is nullable, it has become standard to instead use Optional<> to express this at an API level. While it doesn't completely protect against NullPointerExceptions (aka the "billion dollar mistake"), it's a step in the right direction, and some IDEs will even log warnings if you assign null to a normal, non-Optional type.
In the case of TypeScript, a --noImplicitVoidable flag could be added in the distant future, which would make all types non-voidable (that is, you could not assign null or undefined to them) by default. While this is a major breaking change, it can be mitigated by having "void-protected" types in the language for some time prior. That way, most code will already be using ? prefixes wherever void types are expected.
Finally, under this proposal, there are basically four levels of voidability, which is more relevant if this hypothetical --noImplicitVoidable flag comes to be. A lot of this has been adapted from @tejacques's response in the non-nullable thread:
interface Foo {
  w: string;
  x: ?string;
  y?: string;
  z?: ?string;
}

// alternately
function foo(w: string, x: ?string, y?: string, z?: ?string): void

w: string behaves exactly like it does currently: it must be explicitly set, but it can be void. Under --noImplicitVoidable, it would be a strong guarantee of existence.
x: ?string is new: it must be explicitly set, but can be void. As is, it behaves exactly like w, but provides more safety for a consumer.
y?: string continues to behave as-is. Under --noImplicitVoidable, though, it would probably be treated as "undefinable" - that is, it may have a value of undefined (implicitly, if it's not set) and accessing it must be guarded, but it would never have a value of null.
z?: ?string is equivalent to y, but again provides more safety for consumers and would be preferred over y in most new code.

Hey @dallonf, thanks for taking the mantle on this and bringing it back. I'm not sure what the TypeScript teams wants to do with this, but since the last discussion about it took place, there has been a lot learned and changed in the language itself.
I think this is the right way forward, but there is a lot of trickiness surrounding --noImplicitVoidable that needs to be ironed out. I still think it makes sense to tackle the problem as multiple separate ones.
I'm going to repeat a lot of what you just said, but splitting it up into what I see as the logical separations just to get the full picture and why this particular issue is important and related to it.
It's important to note that only the first issue below is encompassed by this proposal. I'm only rehashing the rest of the steps here so that they can be further broken out into new proposals in the future, or discussed here as appropriate.
First Problem - built-in Option type or Void-protected types (no breaking changes -- this proposal!)
This is the ?T type, which internally to the TypeScript compiler would be represented as the type T | void.
This can actually be done right now with a type alias and custom user type-guard, and it works well, but IMO it's not enough and would be much easier and more idiomatic JavaScript to work as a built-in.
Secondly, the optional argument type, x?: string should really be internalized as the currently un-specifiable x | undefined. This simply ensures that a typeguard needs to be used to access it, but it is a different contract than the option type. Specifically, it is either well defined to be of T or it is undefined, it cannot be null. I actually think continuing to use this syntax is preferable to x?: ?string when applicable, because you are saying that you optionally take a non-voidable argument. So if the argument is supplied, it cannot be void.
In order to use an option type variable: x: ?string as a string it must have a type guard.
For any non-generic type other than ?number, the following acts as the appropriate type-guard:
let x: ?string;
if (x) {
  // x is string
}

This is essentially as elegant as it gets. It's very idiomatic javascript, it's concise, and the intention is clear.
For ?number, or ?T:
let x: ?T
if (x != null) {
  // x is T
}

That guard is not nearly as beautiful as if(x), but most of the time it should be enough. Alternatively it could allow if(x) until it is known that T is a number type, and then give an error. That is more flow-like.
Other guards: ||
It is very idiomatic JavaScript to say something like:
x = x || "fallback";

This should implicitly unwrap the option, so if x had been a ?string, now x is a string. Had x been some other type, like object, then you would get a union: object | string.
More concretely: ?T || U -> T | U.
This has the same caveat for numbers, where the number 0 fails the check, and as such it's not the most appropriate. Nonetheless, it's still an idiom used in JavaScript, and should work the same way type-wise.
Future possibility (Not necessary for any of this to work, but nice to think about): Elvis operator ?.
I think this would make a great future addition to JavaScript in general, and if it does make it, it will make dealing with options very smooth and elegant. That said, I'm only including it here to demonstrate how Options can/will be used and integrated.
The ?. operator would only be available on ?T types, and it would behave exactly as it does in C# and other languages:
This operator will become more important as will be denoted later.
let x: ?string;
let y = x
  ?.substring(1)
  ?.toUpperCase(); // y: ?string

Specifically, if a property or method exists, it uses that property or method, and if it does not, returns null. If the resulting type of that property or method was a U, it becomes a ?U. This means the Elvis operator is exactly equivalent to syntactic sugar over a traditional Option.map or Option.bind.
Second Problem -- Ensuring a variable has been initialized (breaking change, depends on Option types)
This is a really really tough problem. How do you ensure that non-null variables have all been initialized before use? There are some more straightforward things that can be done which cover many cases, but as always it's the corner cases which are difficult:

Observed assignment
Constructor must set/initialize all non-nullable properties
Array construction

If a function can return null or undefined, it should have a return type of ?T.
This is unfortunately where a lot of breaking changes will be required. Specifically in regards to API. There are already loads of existing constructors and functions which do not adhere to this, whose APIs would need to change to return an Option variant. However, this is something that could happen slowly over time. TypeScript can explicitly state that non-null types signatures in code it does not have the source for cannot be enforced, and it is up to the user to ensure they have the proper definitions. The typings in lib.d.ts can also be updated over time to reflect this as well -- each of which will be a breaking change, but in a very minor way. I think this is the hardest part to swallow about the entire approach, but this is substantially mitigated by allowing assignment of non-voidable variables to option types (when --noImplicitVoidable is not given).
For Arrays specifically, this can be mitigated by changing the type of Array<T>(n: number) from T[] to ?T[]. (Note: the type for ?Array<T> would be shorthanded to T?[], ?T[] implies Array<?T>). This can be upgraded to a proper T[]by looping over it and upgraded the assigned type at every entry. Array literals[], the numberless array constructor, and the arguments constructor would not need to return ?T[]` as they are well specified.
Another issue is something like JSON.parse<T>(jsonStr) which currently returns a T, but there's no actual guarantee at all that's what you get back. Really it should be changed to return a ?T. The issue here is that if you check that the result is not null, it's upgraded to a T, but you have no guarantee whatsoever this is reflective of reality.
As such, the best TypeScript can do is provide a weak guarantee that non-voidable types are actually non-void. In my opinion this is OK, as it will be correct the vast majority of the time, and will improve as library authors improve and correct their APIs with the option type.
This is also where the Elvis operator can help (in the hopeful future). Because the Elvis operator automatically returns the property as an option type, it realistically represents the true interaction with the result:
type ParseType = { some: { thing: string } };
let parsed = JSON.parse<ParseType>("null"); // ?ParseType
les res = parsed?.some?.thing || "fallback"; // string
//     ?ParseType -> ?{ thing: string } -> ?string || string -> string

Third Problem -- --noImplicitVoidable (breaking change, requires Second Problem)
So far, the above can be accomplished without any need for --noImplicitVoidable. However, when it is added, the behavior is as follows:

Assignment of void/undefined to any non-option type is an error.
Use of a non-void variable before initialization is an error
Option types cannot be implicitly assigned to non-void types (cannot assign a ?T to a T), must use the || operator or a type guard if/else.
Only checks/verifies this for the source code it compiles

This is a fairly obvious breaking change as code which previously worked will no longer work, but because it is optional, and very useful, it feels right.
This again gives the general weak guarantee that regular types are not void, with the same caveat that external definitions may not adhere to this contract. However, the more the definitions mature the better adherence to this contract the language will have. This flag changes this from simply a convention into something that is enforced for your own code.
This is a simplification from the suggestion in the previous thread, as it doesn't specify doing any kind of enhanced analysis beyond trying to check/ensure initialization (second point).
***NOTE: The Elvis operator is not required for this proposal, it is only used here to demonstrate a possible future. Any discussion about it should happen either in another issue or as a TC39 / EcmaScript proposal.

@tejacques nit:

For any non-generic type other than ?number, the following acts as the appropriate type-guard:
let x: ?string;
if (x) {
  // x is string
}

Unfortunately ?string and ?boolean couldn't be guarded this way, since empty strings and false are also falsy.

@yortus: well, technically it could, right? If x: ?string is truthy, then you are absolutely sure it is a string. The catch is that if (x) would not detect empty strings, which makes it a bit of an antipattern. (unless your logic mirrors the isNullOrEmpty concept common in C# and Java)

As I understood it, if(x) was proposed as a guard only against null and undefined. For that purpose it must certainly not fail on 'non-void' values that are valid in the domain of their respective types, including zero, the empty string, and false. But it's a nit, not a showstopper - as @tejacques points out there is the more reliable if (x != null) guard, which would need to be used for ?number, ?string and ?boolean among others.

@yortus: You're correct about the guard. If it was going to be special cased, it should probably only be on object types. I'll update my post to reflect this, but special casing at all might be the wrong thing to do (wouldn't work in generics, and makes the developer need to be aware of context).

closing this in favor of https://github.com/Microsoft/TypeScript/pull/7140


Handle rate limiting

Hubtty may receive 403 http responses from github, however it does not properly handle rate limiting.
Exception: Received 403 status code: {"message":"API rate limit exceeded for user ID 225909.","documentation_url":"https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"}

It should throttle back and retry later. Right now it doesn't retry at all which can cause it to miss some PRs.

I merged a series of commit that should address the github rate-limiting issue. The most important fixes where to take the X-RateLimit-* headers into account and to sync projects in batch to decrease our use of the search API.
There is still room for improvement, where we could use conditional requests. However the current situation should be good enough for now so I'm closing this issue.


Proxy use

Hi,
great work around the api and it is working well.
One think i was thinking to add would be use of proxies via request/session.
Best

done! just specify your https proxy using:
msft = yf.Ticker("MSFT")
msft.history(period="max", proxy="https://<IP_ADDRESS>:1080")

or via:
df = ys.download("MSFT", period="max", proxy="https://<IP_ADDRESS>:1080")

Hi,
Thanks for the update, but it seems that there is an issue due to the renaming..
I can' install it anymore, nor update...
Collecting fix-yahoo-finance
Using cached https://files.pythonhosted.org/packages/a5/f3/aba3b563452faaea71b4161754991e3143751ec9141d83c4bce05992332a/fix-yahoo-finance-0.1.37.tar.gz
ERROR: Complete output from command python setup.py egg_info:
ERROR: Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\%%\AppData\Local\Temp\pycharm-packaging\fix-yahoo-finance\setup.py", line 19, in <module>
    from fix_yahoo_finance import __version__
  File "C:\Users\%%\AppData\Local\Temp\pycharm-packaging\fix-yahoo-finance\fix_yahoo_finance\__init__.py", line 22, in <module>
    import yfinance
ModuleNotFoundError: No module named 'yfinance'
----------------------------------------

ERROR: Command "python setup.py egg_info" failed with error code 1 in C:\Users%%%\AppData\Local\Temp\pycharm-packaging\fix-yahoo-finance\

Hi
Just use:
pip install yfinance

And then use:
import yfinance as yf

# do stuff here
...

On 26 May 2019, at 19:13, nono-london<EMAIL_ADDRESS>wrote:
Hi,
Thanks for the update, but it seems that there is an issue due to the
renaming..
I can' install it anymore, nor update...
Collecting fix-yahoo-finance
Using cached
https://files.pythonhosted.org/packages/a5/f3/aba3b563452faaea71b4161754991e3143751ec9141d83c4bce05992332a/fix-yahoo-finance-0.1.37.tar.gz
ERROR: Complete output from command python setup.py egg_info:
ERROR: Traceback (most recent call last):
File "", line 1, in
File "C:\Users%%\AppData\Local\Temp\pycharm-packaging\fix-yahoo-finance\setup.py",
line 19, in
from fix_yahoo_finance import version
File "C:\Users%%\AppData\Local\Temp\pycharm-packaging\fix-yahoo-finance\fix_yahoo_finance_init_.py",
line 22, in
import yfinance
ModuleNotFoundError: No module named 'yfinance'
ERROR: Command "python setup.py egg_info" failed with error code 1 in
C:\Users%%%\AppData\Local\Temp\pycharm-packaging\fix-yahoo-finance\
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/ranaroussi/yfinance/issues/66?email_source=notifications&email_token=AAJBNMVKWKPZR5KIDDRP6STPXKZJDA5CNFSM4HPVINQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWIIZ7I#issuecomment-496012541,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAJBNMTYPADN6U4PDJ4HDQTPXKZJDANCNFSM4HPVINQA
.

Hi,
Not sure how to do this in pucharm. Installing is usually done via the ide
not pip.
As well your peogram still shows as fix-yahoo in it. Maybe they will update
the repository later.
Thanks
On Sun, 26 May 2019 at 18:18, Ran Aroussi<EMAIL_ADDRESS>wrote:

Hi
Just use:
pip install yfinance

And then use:
import yfinance as yf

# do stuff here
...

On 26 May 2019, at 19:13, nono-london<EMAIL_ADDRESS>wrote:
Hi,
Thanks for the update, but it seems that there is an issue due to the
renaming..
I can' install it anymore, nor update...
Collecting fix-yahoo-finance
Using cached
https://files.pythonhosted.org/packages/a5/f3/aba3b563452faaea71b4161754991e3143751ec9141d83c4bce05992332a/fix-yahoo-finance-0.1.37.tar.gz
ERROR: Complete output from command python setup.py egg_info:
ERROR: Traceback (most recent call last):
File "", line 1, in
File
"C:\Users%%\AppData\Local\Temp\pycharm-packaging\fix-yahoo-finance\setup.py",
line 19, in
from fix_yahoo_finance import version
File
"C:\Users%%\AppData\Local\Temp\pycharm-packaging\fix-yahoo-finance\fix_yahoo_finance_init_.py",
line 22, in
import yfinance
ModuleNotFoundError: No module named 'yfinance'
ERROR: Command "python setup.py egg_info" failed with error code 1 in
C:\Users%%%\AppData\Local\Temp\pycharm-packaging\fix-yahoo-finance\
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
<
https://github.com/ranaroussi/yfinance/issues/66?email_source=notifications&email_token=AAJBNMVKWKPZR5KIDDRP6STPXKZJDA5CNFSM4HPVINQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWIIZ7I#issuecomment-496012541

,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AAJBNMTYPADN6U4PDJ4HDQTPXKZJDANCNFSM4HPVINQA

.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ranaroussi/yfinance/issues/66?email_source=notifications&email_token=AB245CPRU56TSNISG7W7VMDPXKZ47A5CNFSM4HPVINQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWII5AY#issuecomment-496012931,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB245CKYLMYE3FLPLDHNMPDPXKZ47ANCNFSM4HPVINQA
.

hi,
do you think you can pic up as well the company financial's?
for instance:
https://finance.yahoo.com/quote/JMIA/financials?p=JMIA
best
On Sun, 26 May 2019 at 18:53, Nono Dup<EMAIL_ADDRESS>wrote:

Hi,
Not sure how to do this in pucharm. Installing is usually done via the ide
not pip.
As well your peogram still shows as fix-yahoo in it. Maybe they will
update the repository later.
Thanks
On Sun, 26 May 2019 at 18:18, Ran Aroussi<EMAIL_ADDRESS>wrote:

Hi
Just use:
pip install yfinance

And then use:
import yfinance as yf

# do stuff here
...

On 26 May 2019, at 19:13, nono-london<EMAIL_ADDRESS>wrote:
Hi,
Thanks for the update, but it seems that there is an issue due to the
renaming..
I can' install it anymore, nor update...
Collecting fix-yahoo-finance
Using cached
https://files.pythonhosted.org/packages/a5/f3/aba3b563452faaea71b4161754991e3143751ec9141d83c4bce05992332a/fix-yahoo-finance-0.1.37.tar.gz
ERROR: Complete output from command python setup.py egg_info:
ERROR: Traceback (most recent call last):
File "", line 1, in
File
"C:\Users%%\AppData\Local\Temp\pycharm-packaging\fix-yahoo-finance\setup.py",
line 19, in
from fix_yahoo_finance import version
File
"C:\Users%%\AppData\Local\Temp\pycharm-packaging\fix-yahoo-finance\fix_yahoo_finance_init_.py",
line 22, in
import yfinance
ModuleNotFoundError: No module named 'yfinance'
ERROR: Command "python setup.py egg_info" failed with error code 1 in
C:\Users%%%\AppData\Local\Temp\pycharm-packaging\fix-yahoo-finance\
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
<
https://github.com/ranaroussi/yfinance/issues/66?email_source=notifications&email_token=AAJBNMVKWKPZR5KIDDRP6STPXKZJDA5CNFSM4HPVINQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWIIZ7I#issuecomment-496012541

,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AAJBNMTYPADN6U4PDJ4HDQTPXKZJDANCNFSM4HPVINQA

.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ranaroussi/yfinance/issues/66?email_source=notifications&email_token=AB245CPRU56TSNISG7W7VMDPXKZ47A5CNFSM4HPVINQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWII5AY#issuecomment-496012931,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB245CKYLMYE3FLPLDHNMPDPXKZ47ANCNFSM4HPVINQA
.

done! look at the updated readme

@ranaroussi Hi, proxy only works if threads sets to true, is this expected? Thanks!


Components seem too big

I'm starting with react-toolbox, is a awesome library, thanks for that.
One thing that I can't understand, is that the size of components are so big in my project.
See comparison with the showcase components.

My sample project is here.
What I'm doing wrong?

Sorry, just adding import 'react-toolbox/lib/commons.scss'; soved my problem.

Is there a way to fix the sizing without including commons.scss? I don't want to interfere with the font size of the rest of my app.

Yes, you can se the default unit to 10px. It should work for the most recent versions. Anyway I encourage you to use the latest beta which gets rid of commons.scss so it doesn't mess with your setup at all.

Awesome!
Thanks for the prompt reply @javivelasco, will try out the beta.

@javivelasco I can't be bothered to figure out the changes to my webpack right now to move to the beta. To your first point, how does one set the default unit?

Sorry for this 3 days response @godd9170. The builds now are produced with all css transformed except for the postcss-variables plugin which also included in css-next. What you need to change is the CSS loader. Now instead of using SASS loader + Postcss you just need postcss with one single plugin. To get yourself oriented you can check the webpack configuration in the package (it is used to display a showcase for development purposes). You may need less config, only css-next is required indeed.
Also you may want to try out react-toolbox-themr which allows you to forget about messy setups. The default unit is 10px by default now so we don't mess with rems. You can still set it to 1rem if your body size is 16px as it was with commons.scss.


Export Traces

May as well let people export their traces as well.

Implemented https://github.com/icirellik/glowforge-material-manager/pull/69


I can't get the recycleOnFileChange element in web.config to work

I want IIS to restart my asp.net core app if my appsettings.myEnvironment.json file changes.  I think this should be possible using the recycleOnFileChange element in my web.config file as follows:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <system.webServer>
    <handlers>
      <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModule" resourceType="Unspecified" />
    </handlers>
     <aspNetCore processPath="%LAUNCHER_PATH%" arguments="%LAUNCHER_ARGS%" stdoutLogEnabled="false" stdoutLogFile=".\logs\stdout" forwardWindowsAuthToken="false">
        <recycleOnFileChange>
           <file path="d:\myWebApplicationPath\appsettings.myEnvironment.json"/>
        </recycleOnFileChange>
     </aspNetCore>
  </system.webServer>
</configuration>

Unfortunately this doesn't seem to have any effect.  I can't find much in the way of documentation or samples for using recycleOnFileChange.  Should it be working?  If so, what am I doing wrong?
Many thanks
Bill

@shirhatti has implied that recycleOnFileChange is nop in https://github.com/aspnet/Docs/issues/1965

Yeah. recycleOnFileChange currently just no-ops. The element remains a part of the schema to prevent us from breaking existing applications.

@BillDines reloadOnChange won't help?
.AddJsonFile("appsettings.json", reloadOnChange: true)

Oh well, that explains it!  @GuardRex - I could probably implement what I need with reloadOnChange, its just much harder work.  I am using the Options pattern and I think I'd have to do quite a lot of work to refresh my options objects after reload - something I'd hoped to avoid!

Thanks @GuardRex - I had seen you can use a callback.  I'll see if I can do something in the callback that works nicely with DI and IOptions...

It would be nice to update the docs @danroth27

@muratg Could you please open an issue in the aspnet/Docs repo with the details of what you want added to the docs?

It's in the section at https://github.com/Microsoft/Docs-1/blob/master/aspnet/aspnetcore/fundamentals/configuration.md#using-the-built-in-sources
It only says that ...

When specifying files as configuration sources, you can optionally specify whether changes to the file should result in the settings being reloaded. This is configured by passing in a true value for the reloadOnChange parameter when calling AddJsonFile or similar file-based extension methods.

https://github.com/aspnet/Docs/issues/1990 tracking doc work

We are closing this issue because no further action is planned for this issue. If you still have any issues or questions, please log a new issue with any additional details that you have.


Fixed a bug and added x-Days.

If Poloniex return a error the bot shuts down, i have added a 60 sec delay before checking again.
The bot will run longer before hitting a error now.

X-days is a good idea.
But for the error, not sure how can it happen. Will have to look at it closer. My bot literally runs over a month without restart, without a problem. Perhaps you use Windows?

I use a raspberry pi, and it randomly gets to a error while running. I'm not 100% sure about it. It's says that my keys are invalid.

Don't merge yet, my branch have some bugs.

Lately my bot has been catching Poloniex errors frequently. It seems their API sometimes gets overloaded. Glad to see someone implemented a check so I don't have to :P

Please check another branch which is more robust: https://github.com/Mikadily/poloniexlendingbot/tree/multithreading

A timeout on requests has been added as part of this pull requests:
https://github.com/Mikadily/poloniexlendingbot/pull/69
I think this pull request needs to be rebased on the current master - with only the xdays change included and extended to include also command line parameter.

I like the x-day feature.

I have made a new pull request. #108
Look at that one instead of this.


PR for code review - Redux setup

This PR features Redux implemented with Vite+React+Tailwind for the bookstore app
Features

Added Redux store
Added uuid for unique identification generation
Added folder structure following Ducks pattern

Live view

@wickathou I am very sorry, I initially approved it but I forgot about the MIT, you can ask for a second opinion or you request for review again. My mistake, my apologies.


Suggested fix for issue 1067: add 'nonzero_occurrences' to solve igno…

…red Resolves for status history like: 1,1,1,2,0. https://github.com/sensu/sensu/issues/1067
Requires a PR for sensu-plugin as well (same title)

Or should we store the timestamp for the last OK status? Then no time calculation w/ counts and interval are required. What do you think? 'last_ok'?

Ok, my "patches" are awful, they only make stuff worse.  What about an "alert_triggered" boolean, which gets set once occurrences is reached, and only unset by a 0?

Please ignore this.  It is wrong.


A issue of using Mono Debug to debug a example with gui.

VSCode Version: Code 1.9.1 (f9d0c687ff2ea7aabd85fb9a43129117c0ecf519, 2017-02-08T23:44:55.542Z)
OS Version: Linux x64 3.19.0-80-generic
Extensions:

Extension
Author
Version

cpptools
ms-vscode
0.10.1

csharp
ms-vscode
1.7.0

mono-debug
ms-vscode
0.15.3

Steps to Reproduce:

Use mono-debug to debug a c# example with gui, it works.
The debug feature doesn't work(can't step over or step into to watch variables) when the gui of example showing.
Could you help check?

@tengxianbing how does your launch configuration look like?

This issue was moved to Microsoft/vscode-mono-debug#24


Missing template task for some environments

I've just noticed an issue with Poseidon that needs further investigation:
According to CodeOcean, some environments were not found when executing code and were required to be synced CODEOCEAN-11J. Indeed, CodeOcean still shows empty pools for some environments, including the IDs 11, 18, 22, 33:

Nomad, however, has enough jobs scheduled:

During my investigation, I already identified that Poseidon is not aware of the environments:
{
    "executionEnvironments": [
        {
            "prewarmingPoolSize": 5,
            "cpuLimit": 20,
            "memoryLimit": 512,
            "image": "openhpi/co_execenv_java:17",
            "networkAccess": false,
            "exposedPorts": null,
            "id": 31
        },
        {
            "prewarmingPoolSize": 5,
            "cpuLimit": 20,
            "memoryLimit": 512,
            "image": "openhpi/co_execenv_java:8-antlr",
            "networkAccess": false,
            "exposedPorts": null,
            "id": 10
        },
        {
            "prewarmingPoolSize": 5,
            "cpuLimit": 20,
            "memoryLimit": 256,
            "image": "openhpi/co_execenv_r:4",
            "networkAccess": false,
            "exposedPorts": null,
            "id": 28
        },
        {
            "prewarmingPoolSize": 5,
            "cpuLimit": 20,
            "memoryLimit": 512,
            "image": "openhpi/co_execenv_julia:1.8",
            "networkAccess": false,
            "exposedPorts": null,
            "id": 34
        },
        {
            "prewarmingPoolSize": 2,
            "cpuLimit": 20,
            "memoryLimit": 256,
            "image": "openhpi/co_execenv_python:3.4",
            "networkAccess": false,
            "exposedPorts": null,
            "id": 14
        },
        {
            "prewarmingPoolSize": 2,
            "cpuLimit": 20,
            "memoryLimit": 256,
            "image": "openhpi/co_execenv_ruby:2.5",
            "networkAccess": false,
            "exposedPorts": null,
            "id": 25
        },
        {
            "prewarmingPoolSize": 15,
            "cpuLimit": 20,
            "memoryLimit": 256,
            "image": "openhpi/co_execenv_python:3.8",
            "networkAccess": false,
            "exposedPorts": null,
            "id": 29
        },
        {
            "prewarmingPoolSize": 2,
            "cpuLimit": 20,
            "memoryLimit": 256,
            "image": "openhpi/co_execenv_python:3.7-ml",
            "networkAccess": false,
            "exposedPorts": null,
            "id": 30
        }
    ]
}

This matches Nomad as well:

Hence, Poseidon thinks that everything is okay and is not issuing a Prewarming Pool Alert:

Poseidon, however, was not restarted for quite a while:
root@poseidon-terraform:/home/ubuntu# systemctl status poseidon
● poseidon.service - Poseidon
     Loaded: loaded (/etc/systemd/system/poseidon.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2023-12-08 02:50:23 UTC; 18h ago
   Main PID: 623 (poseidon)
      Tasks: 9 (limit: 4647)
     Memory: 291.7M
        CPU: 2h 48min 12ms
     CGroup: /system.slice/poseidon.service
             └─623 /usr/local/bin/poseidon

Probably, the affected environments were lost during the night:

The Sentry Issue CODEOCEAN-11J does no longer exist. @MrSerth have you experienced this issue lately or is there a follow-up Sentry issue?
From your images, we can derive that Poseidon restarted at 2023-12-08 02:50:23 UTC. According to the Grafana image, even after the restart, environment 33 was still active. It got inactive at 03:50. The Nomad Job was no longer running.
Indeed, when Poseidon is notified about a stopped environment job, it does absolutely nothing.
https://github.com/openHPI/poseidon/blob/8390b90fe21e393f96421ae5ca8d7f415e7b680c/internal/runner/nomad_manager.go#L307-L309
However, the environment is only removed from Poseidon's memory when requested via the API. If that is not the case, I have to assume that the Grafana image uses a different timezone and the environment stopped existing once Poseidon got restarted.
How should we deal with this issue?

Rely on Nomad

Nomad should be responsible for correctly restarting and rescheduling the environment
We have introduced changes in between that might reduce the likelihood of failures (#611)
The effects of a lost environment are not too bad

CodeOcean recreates the environment once a user wants to access it

Poseidon handling

Just Logging
Recreation

@MrSerth have you experienced this issue lately or is there a follow-up Sentry issue?

I linked the follow-up Sentry issue CODEOCEAN-12Y, which I triggered manually on our staging system (so no real issue). But still, it would be the one where further events are grouped.
Besides that, I don't have another recent occurrence, but saw the potentially lost environment on CodeOcean's dashboard during the past student courses (so in spring 2024).

If that is not the case, I have to assume that the Grafana image uses a different timezone and the environment stopped existing once Poseidon got restarted.

I am not sure about the time zone (sorry!), but would assume that the Grafana dashboard is shown in my browser's time zone (should be CET, +01:00) -- that's at least the case when I visit Grafana today (and I cannot remember that it was UTC).
Otherwise, I agree with your assumption: Probably, Poseidon got restarted after the environment job got lost, hence the recovery did not recover the environment at all.

The effects of a lost environment are not too bad

CodeOcean recreates the environment once a user wants to access it

That's true, of course. Still, if we can prevent this error, it would be even better.

How should we deal with this issue?

Good question. I like that we improved the situation already through the better restart and rescheduling mechanism. Still, jobs might get lost. Hence, if an environment job fails finally (and is no longer retried by Nomad), we could (potentially) restart it, couldn't we? Do we have all the required information to do so? And would you assume restarting is fine, or could this result in duplicated jobs, ...?
As an additional precaution, I tend to add support to Poseidon for this case, too (depending on your answer to the previous questions).

Hence, if an environment job fails finally (and is no longer retried by Nomad), we could (potentially) restart it, couldn't we? Do we have all the required information to do so?

We could and we have all the required information to do so. However, it would add some new complexity in the Nomad Event Stream Handling and new workflows within Poseidon.

Would you assume restarting is fine

There are three cases why environment jobs fail finally:

Because Poseidon stopped it

All good (as long as we are not aware of Poseidon deleting environments without a respective API request.)

Because Nomad stopped it as defined in the restart and rescheduling policy

Likely this is a misconfigured environment (e.g. wrong image)

All good
In this case, Poseidon should not restart the environment job, because we intended to not infinitely restart (invalid) environments/jobs

We should validate each event of this issue to catch other causes of stopped environments.

Because Nomad stopped it unexpectedly

We should monitor if this case happens and hunt down each cause.

All in all, it feels like we would re-implement too much of Nomad's functionality and responsibility. If we want to change the behavior we might rather adjust the job policy. I would go with adding visibility to this issue as proposed with #668.

Hence, if an environment job fails finally (and is no longer retried by Nomad), we could (potentially) restart it, couldn't we? Do we have all the required information to do so?

We could and we have all the required information to do so. However, it would add some new complexity in the Nomad Event Stream Handling and new workflows within Poseidon.
Okay, I see. Especially given the fact that we could try to advise Nomad to restart the job again, let's try this Nomad-based approach first.

There are three cases why environment jobs fail finally

Because Poseidon stopped it

That's fine, and I am not aware of any false deletion requests by Poseidon right now.

Because Nomad stopped it as defined in the restart and rescheduling policy
Likely this is a misconfigured environment (e.g. wrong image)

For the occurrence I created the issue for this (misconfigured environment) is highly unlikely. The environment (command, image, ...) doesn't change very frequently and I cannot remember of real issues with that recently. For sure, the restart and rescheduling limits could have been reached, due to network issues, agent shutdowns, etc.
I get that a Poseidon-based solution might not make sense here, but can we get creative and nevertheless adjust the task policy for environments (not for regular runner jobs) here? For example, I would assume that a wrong image leads to immediate and countless restarts. Other availability issues, however, would only fail for a limited time (like for the duration of one or two restarts) and work before or after that. With a comparatively small interval setting (1h / 30m?) one could try to narrow those case. Another idea would be to set environment stone unlimited restarts, and simply check whether (in Poseidon) whether a first error occurs shortly after synchronizing the environment (in which case it might be erroneous). Or ... some other idea you might have?

Because Nomad stopped it unexpectedly

Yes, we should monitor these and I like the enhanced logging with #668.

Since the changes introduced in #668 were deployed, we saw the first event in https://github.com/openHPI/poseidon/issues/672. Let's double check this issue, too.

Because Nomad stopped it as defined in the restart and rescheduling policy

#668 monitors also these cases. We might want to wait for an occurrence to verify that this is still a real issue.

A misconfigured environment is highly unlikely. The environment (command, image, ...) doesn't change very frequently and I cannot remember of real issues with that recently. For sure, the restart and rescheduling limits could have been reached, due to network issues, agent shutdowns, etc.

That's right, I agree

Can we adjust the task policy for environments (not for regular runner jobs) here?

Yeah, we might overwrite some policies within the Poseidon Code (e.g. just for template jobs).
All in all, it feels like we would spend too much time and code complexity to something without a big impact and without known recent cases. Let's continue investigating this issue, once a real case occurs again.

I would still like to see some improvement here and think adjusting some values for environments could be useful. However, giving the other pending issues (and especially #612), we decided to close and postpone this issue for now.


Customizing AtomCompare

During MCS search is there a way to extend the AtomCompare option to include more sophisticated matching criterion. In my specific case I would like two atoms to match only if their partial charge difference is lower then a given threshold.
Or if there isn't, then how would I go about implementing something like this?

The exact property to compare is _TriposPartialCharge.

This is a great suggestion. Making it easy to add new atom-atom or bond-bond comparison functions would be really helpful.
It's not easy to do this (and can't really be done at all without doing C++ work), but it's something I will try and find some time to do.
There's a connection here to #509

Writing code for a generic algorithm that can take any property is probably hard as you said, but implementing this specific comparison (_TriposPartialCharge) should be easier, right? If yes, where in the code could I add this?


[SWARM-548] Add support for export=true in module.conf

I'll review, merge and release this first thing this morning.
Bob
--
Sent from my iPhone
On Jul 7, 2016, at 5:34 AM, Thomas Diesler<EMAIL_ADDRESS>wrote:

You can view, comment on, or merge this pull request online at:
https://github.com/wildfly-swarm/wildfly-swarm-fraction-plugin/pull/8
Commit Summary

[SWARM-548] Add support for export=true in module.conf

File Changes

M src/main/java/org/wildfly/swarm/plugin/ModuleGenerator.java
https://github.com/wildfly-swarm/wildfly-swarm-fraction-plugin/pull/8/files#diff-0
(8)

Patch Links:

https://github.com/wildfly-swarm/wildfly-swarm-fraction-plugin/pull/8.patch
https://github.com/wildfly-swarm/wildfly-swarm-fraction-plugin/pull/8.diff
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/wildfly-swarm/wildfly-swarm-fraction-plugin/pull/8,
or mute
the thread
https://github.com/notifications/unsubscribe/AAA-TwEfODubMJmwHPfe8YRq5opUGfVfks5qTMg0gaJpZM4JG5yB
.


Feature Request: Specify a folder to import to

Hey, I love your tool.  I'd like to make a suggestion, if I had the option of selecting which folder or subfolder to import my actor to, that would greatly simplify my life.  I've got a billion subfolders, and clicking and dragging each actor can be tedious.  (Not as tedious as creating a new stat block from scratch, mind you).
Thanks for doing what you do!

Ask and you shall receive. Update to version 1.6 and you'll see the new option.


I am trying to develop a chat application with estimated 1 million concurrent users, I saw this can be done by adding multiaple instances of azure pub/sub service. and use connection sharding in negotiate phase. any thoughts about it??

Yes, this is doable using multiple instances. Please do contact our support team if you'd like further technical support. You can also directly contact our product team through awps(at)microsoft.com.


gtk: Implementable implementation for AccessibleText

I don't have the time to review this right now but two questions:

is it important for you to have this in the next release or waiting till a future .x would be okay?
do you think you could add an example of how it could be used?

Maybe two wrapper objects for (range, attr, value) triples wouldn't be such a bad idea. When writing the example it was very cumbersome to change attributes, adding attributes by constants, etc.

I think we can do that as a second step :) but yes, having to deal with constants is not ideal. But also not sure how we can do something that wouldn't be too far away from the original API.
Maybe we can make those attributes an enum? then you would only have a Vec<(Range, TextAttribute)>


BUG?

Amazing work. But may I ask how to solve the following problem?

I first downloaded the pre-training weights, such as gpt2-large and opt-1.3B. Basically unchanged, except that the model is loaded with the downloaded weights.The previous output log is as follows

env:
torch1.12cu113,transformers4.26.0

Yes!
Yes!

same error.
NVIDIA-SMI 515.43.04    Driver Version: 515.43.04    CUDA Version: 11.7
torch-1.13.0+cu117
pytorch-cuda              11.7

in reward json file， the score is None, so i change it into "0.01".

Hey @lonelydancer, as far as I can tell, the reward dataset doesn't play a role in actor pre-training. So, I don't think changing the score value to "0.01" is what caused the assertion error

I also got the assertion error.
Traceback (most recent call last):
  File "/home/anirudh/rlhf/artifacts/main.py", line 51, in <module>
    actor_trainer.train()
  File "/home/anirudh/chatllama_test/venv/lib/python3.10/site-packages/chatllama/rlhf/actor.py", line 373, in train
    est_output = self.model(training_input, attention_mask)
  File "/home/anirudh/chatllama_test/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "<@beartype(chatllama.rlhf.actor.ActorModel.forward) at 0x7f30171c2290>", line 51, in forward
  File "/home/anirudh/chatllama_test/venv/lib/python3.10/site-packages/chatllama/rlhf/actor.py", line 114, in forward
    model_output = self.model.forward(
  File "/home/anirudh/chatllama_test/venv/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 932, in forward
    outputs = self.model.decoder(
  File "/home/anirudh/chatllama_test/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/anirudh/chatllama_test/venv/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 639, in forward
    attention_mask = self._prepare_decoder_attention_mask(
  File "/home/anirudh/chatllama_test/venv/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 542, in _prepare_decoder_attention_mask
    expanded_attn_mask = _expand_mask(attention_mask, inputs_embeds.dtype, tgt_len=input_shape[-1]).to(
  File "/home/anirudh/chatllama_test/venv/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 89, in _expand_mask
    expanded_mask = mask[:, None, None, :].expand(bsz, 1, tgt_len, src_len).to(dtype)
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

This repo is super cool and it would be great if this small err is fixed!

^ update (just for more context)- load_model_test works! I believe this method uses a HF GPT2 tokenizer. However, if I use the load_model method, I get this assertion error. I also tried the initial version of llama_model.py (the one with a test flag) and ran into the assertion error problem when I set test to False (basically the same as calling load_model)

I think this is the dataset/tokenizer problem.
I use the llama tokenizer and opt-1.3b model, get this error at index=815/816.
So what tokenizer should I use for opt-1.3b ?
The data:
[{"user_input": "New research suggests that metrics that are used to measure academic success, such as the number of publications, number of citations, and impact factor, have become targets and follow Goodhart\u2019s Law, according to which, \u201cwhen a measure becomes a target, it ceases to be a good measure.\u201d The original study can be found here: https://academic.oup.com/gigascience/article/8/6/giz053/5506490  Abstract:  **Background**  > The academic publishing world is changing significantly, with ever-growing numbers of publications each year and shifting publishing patterns. However, the metrics used to measure academic success, such as the number of publications, citation number, and impact factor, have not changed for decades. Moreover, recent studies indicate that these metrics have become targets and follow Goodhart\u2019s Law, according to which, \u201cwhen a measure becomes a target, it ceases to be a good measure.\u201d  **Results**  > In this study, we analyzed >120 million papers to examine how the academic publishing world has evolved over the last century, with a deeper look into the specific field of biology. Our study shows that the validity of citation-based measures is being compromised and their usefulness is lessening. In particular, the number of publications has ceased to be a good metric as a result of longer author lists, shorter papers, and surging publication numbers. Citation-based metrics, such citation number and h-index, are likewise affected by the flood of papers, self-citations, and lengthy reference lists. Measures such as a journal\u2019s impact factor have also ceased to be good metrics due to the soaring numbers of papers that are published in top journals, particularly from the same pool of authors. Moreover, by analyzing properties of >2,600 research fields, we observed that citation-based metrics are not beneficial for comparing researchers in different fields, or even in the same department.  **Conclusions**  > Academic publishing has changed considerably; now we need to reconsider how we measure success.", "completion": "This \u201cstudy\u201d is a hot steaming pile garbage and certainly doesn\u2019t show what it claims to. A couple overarching massive flaws:  1. They don\u2019t compare changes in # of papers or # of coauthors over time to changes in population. Their plot of # of papers looks a lot like a plot of the # of people in the world. At no point in the paper do they control for the effect of the increased number of researchers. At no point do they address the sharp decline in the past 15 years. 2. They make no effort to measure the quality of a paper, and yet claim that people are optimizing for metrics instead of improving. You need to actually give an indication that the quality of research isn\u2019t increasing to show that. 3. Their data analysis practice is abysmal. They misleadingly cut axes and rescale plots. They don\u2019t measure correlation, statistical significance, or effect size. They don\u2019t discuss the fact that the trends they\u2019re talking about sometimes only exist in some time periods and not others. They put Plot 10 in a paper, which is a horrible abomination that I still can\u2019t figure out how to read.  Now for a point-by-point refutation....  >First, these results support Goodhart\u2019s Law as it relates to academic publishing: the measures (e.g., number of papers, number of citations, h-index, and impact factor) have become targets, and now they are no longer good measures. By making papers shorter and collaborating with more authors, researchers are able to produce more papers in the same amount of time. Moreover, we observed that the majority of changes in papers\u2019 properties are correlated with papers that receive higher numbers of citations (see Fig.\u00a0S13). Authors can use longer titles and abstracts, or use question or exclamation marks in titles, to make their papers more appealing. Thus, more readers are attracted to the paper, and ideally they will cite it, i.e., academic clickbait\u00a0[45]. These results support our hypothesis that the citation number has become a target. Consequently, the properties of academic papers have evolved in order to win\u2014to score a bullseye on the academic target.  Of course the number of citations and coauthors has gone up, there are more researchers and papers than ever before! At no point did you ever present # of papers per author per year, so don\u2019t make any claims about it. I am 100% sure you could get the actual number in one line of code, and the fact that you choose not to when it\u2019s central to your narrative is a massive red flag.  Additionally, you never consider the alternative hypothesis that writing quality of research papers has improved. Many research papers from the early 1900s are awful to read, incredibly dry, tedious, and poorly written. The fact that papers are shorter and more willing to engage in common rhetorical techniques like using questions in titles is a good thing in my mind. Writing papers that people want to read means writing better research papers.  The analysis in this paper in no way demonstrates Goodhart\u2019s Law. To do that you need to show that an increase in metric is not due to an increase in performance.  >It is worth noting that while the study\u2019s results provide evidence that many citation-based measures have become targets, there also may be other factors that influence academic publication trends. For example, the academic hypercompetitive environment itself may prompt an increase in productivity\u00a0[81], hence increasing the number of papers. However, this claim contradicts the findings of Fanelli and Larivi\u00e8re that researchers\u2019 individual productivity did not increase in the past century\u00a0[52]. Nevertheless, it is important to keep in mind that there may be other underlying factors that contributed to the observed results.  Gee, you think? Shame other than this throwaway paragraph you don\u2019t actually seriously consider any alternatives to the narrative you\u2019re peddling.  >Second, we observed that over time fewer papers list authors alphabetically, especially papers with a relatively high number of authors (see Results of Paper Trends section and Figs\u00a04 and\u00a0S5). These results may indicate the increased importance of an author\u2019s sequence number in the author list, which may reflect the author\u2019s contribution to the study. This result is another signal of the increasing importance of measures that rate an individual\u2019s research contribution.  This isn\u2019t something you can meaningfully measure across all fields. Different fields have different conventions. Virtually every mathematics paper in the world today has alphabetical authorship or random authorship. In any event, this is barely a discernible trend in your plot and you made no effort to quantify the effect size or show that it was statistically significant. In the past 40 years, it looks like the needle has barely moved at all in terms of % of papers that have alphabetical authorship (Fig. 4).  >Third, from matching papers to their L0 fields of study, we observed that the number of multidisciplinary papers has increased sharply over time (see Fig.\u00a06). It is important to keep in mind that these results were obtained by matching keywords to their corresponding fields of study. Therefore, these results have several limitations: first, not all papers contain keywords. Second, the dataset may not extract keywords from papers in the correct manner. For example, we found some papers contained keywords in their online version but not in their offline version (see Results of Paper Trends section). It is also possible that in some fields it is less common to use keywords. Therefore, the papers\u2019 keywords may be missing in the datasets, and the presented results may be an underestimate of the actual number of multidisciplinary studies. Nevertheless, we observed a strong trend in increasing numbers of multidisciplinary papers.  This is totally disconnected from the rest of the analysis and never mentioned again. Is it supposed to be meaningful?  >Fourth, from seeing sharp increases in both the maximal and mean number of self-citations (see Results of Paper Trends section and Figs\u00a07,\u00a09,\u00a010, and\u00a0S12), it is clear that citation numbers have become a target for some researchers, who cite their own papers dozens, or even hundreds, of times. Furthermore, we observed a general increasing trend for researchers to cite their previous work in their new studies. Moreover, from analyzing the percentage of papers without citations after 5\u00a0years, we observed that a huge quantity of papers (>72% of all papers and 25% of all papers with \u22655 references) have no citations at all (see Fig.\u00a09). Obviously, many resources are spent on papers with limited impact. The lack of citations may indicate that researchers are publishing more papers of poorer quality to boost their total number of publications. Additionally, by exploring papers\u2019 citation distributions (see Fig.\u00a010), we can observe that different decades have very different citation distributions. This result indicates that comparing citation records of researchers who published papers during different periods can be challenging.  No, it\u2019s not clear that people are deliberately citing themselves to game research metrics. Self-citation will increase with no change in behavior if the number of people per paper increases, which you\u2019ve already shown. Controlling for this is a basic prerequisite for drawing a meaningful inference, and it\u2019s a shame you didn\u2019t do that. It will also increase with increased specialization, since if I write 5 papers on a single topic and you write 5 papers on 5 different topics it is quite reasonable to assume my papers will show more self-citation than yours. You did show that the percentage of interdisciplinary papers is increasing, this might be a good place to reference that fact.  In order to make claims about \u201csome researchers\u201d you need to actually track to see if that\u2019s a stable group of people over time or not. Shame you didn\u2019t bother to do that either. This is data analysis 101, and you\u2019re failing it. You\u2019re also failing it in other ways, such as changing the scale of plots that are being compared and cutting axes off arbitrarily.  What Figure 9 actually shows is that the percentage of papers with no citations in 5 years has plummeted. I don\u2019t understand how they can possibly write this with a straight face. Both of these numbers hit their all-time low between 2000 and today. In 1900 it was ~98% and today it\u2019s ~75%. That\u2019s good, right? Why are you presenting this like it\u2019s a bad thing?", "score": null}]

opt-1.3b is from hugging face:  models--facebook--opt-1.3b

From the official paper:

We tokenize all corpora using the GPT-2 byte level BPE tokenizer (Sennrich et al., 2016; Radfordet al., 2019; Brown et al., 2020). Our final corpus contains roughly 180B tokens.

This might solve your problem:

"facebook/opt-1.3b" in the tokenizer_model_path param (https://huggingface.co/facebook/opt-1.3b)
Change load_model to load_mdel_test in actor.py - this loads in a tokenizer from HF
https://github.com/nebuly-ai/nebullvm/blob/d16dfd8312218540fbdc633222e4655480040778/apps/accelerate/chatllama/chatllama/rlhf/actor.py#L52
https://github.com/nebuly-ai/nebullvm/blob/d16dfd8312218540fbdc633222e4655480040778/apps/accelerate/chatllama/chatllama/rlhf/actor.py#L58

i truncate the input data to the max_length, there is no such error.

@lonelydancer is it max_length number of characters or words?

Hi Everyone! @lonelydancer  @bnuzhanyu  @iMountTai  @AAnirudh07
This problems should have been fixed in the PR #306 .
Let me know if you are still struggling.


[FullyConnected] Added NHWC support for FC_Layer inference part.

This PR contains changes from the commits in PR#2238 and PR#2240 since these are required to evaluate the end to end working by the tests
Commit 75fafa06900f417f9bf8fdca4e17c068d1e7516d is the one that contains the newly added changes.
Self evaluation:
1. Build test:   [X]Passed [ ]Failed [ ]Skipped
2. Run test:     [X]Passed [ ]Failed [ ]Skipped

:memo: TAOS-CI Version: 1.5.20200925. Thank you for submitting PR #2246. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://ci.nnstreamer.ai/.

:octocat: cibot: @adwaith-a, A builder checker could not be completed because one of the checkers is not completed. In order to find out a reason, please go to http://ci.nnstreamer.ai/nntrainer/ci/repo-workers/pr-checker/2246-202308081330410.49277210235596-5f6854ea8476c398ead90d21a4a71d31b7c26bfc/.


can't reproduce 4x scaling

with default params i can't reproduce your psnr and ssim on any of the test sets. did you adjust lr, momentum, decay, etc for 4x case?

@makslevental , no, I haven't, you could try more times

@leftthomas which version of pytorch are you using?

@makslevental at that time, is 0.3

@leftthomas thank you very much :)

@leftthomas did you ever notice the discriminator overpowering the generator? i my g_score get much much smaller than my d_score (over time) and then of course the psnr/ssim never improves. this doesn't happen on 2x but it does on 4x and 8x.

@makslevental this is the common problem of GAN

@leftthomas yes i know. i'm asking you how you overcame it

@makslevental this repo is reimplementation of SRGAN, I just code like the paper said, I haven't try to improve it.


 I think what I had gotten wrong was the manifest.json config change from "*://github.com/*/network/members" to "*://github.com/*/*".
Would you mind explaining why that is necessary ? I thought keeping the matches as restrictive as possible would be the best.

when you go to Dependencies then click on Forks it look like the whole page changed but what actually happened is that :

Pjax fetched only partes of Forks page via AJAX and rendered it without reloading the page layout or any resources (js, css ...)
then it updates the browser's current URL using history.pushstat

i added an event listeners that monitor pjax requests then call init() after it ended.

document.addEventListener('pjax:end', init);

the init() function well check if current URL is "*://github.com/*/network/members" then add css and run initial_request()

if (pathComponents[4] == "members"){...

chrome extensions normally doesnt monitor history.pushstat so when you go to Forks from Dependencies page your extension didn't get enabled cause chrome still think you are in the Dependencies page

if you wanna keep the matches as restrictive as possible you need to add every page that link to Forks page and use pjax
      "matches": [
        "*://github.com/*/*/community",
        "*://github.com/*/*/graphs/*",
        "*://github.com/*/*/network/*",
        "*://github.com/*/*/pulse"
      ],

Okay, you explained it really well, thank you!


update vlad, gmm, kmeans & sift

use fused types for np.float32 and np.float64 support
update test cases for vlad
update documents

Awesome - great work

Minor comment is just that this is the reason the other methods were flipped to avoid the transpose -> because the transpose will cause a copy


Add a Dockerfile

It would be great to have an official Docker image :)

https://hub.docker.com/r/axnux/npm-register/

@razbegaev-nikolay, thank you. But it's not official, not linked to latest master and does not have version tags available.