Feature request: timeout -t

Would it be possible to get the -t switch to specify a timeout for a testcase? Fuzzing is only efficient when the inputs do not take too long. I'm currently trying to do another run with Tika, but alredy the fourth input dry run is just stuck for several minutes...

As you said in another issue:

JQF does not handle hangs at all at this point, which is a limitation (mainly because we are interested in unit tests and not long-running programs). I'm trying to find good ways to kill running threads preemptively without breaking too much (Thread.stop in the JDK is deprecated, and Future.cancel or Thread.interrupt leaks resources because the threads aren't actually killed). If you have any suggestions here, I would love to hear them!

My first suggestion is to try implementing Thread.interrupt as suggested in the docs, for some cases that might help. I would implement this as a first attempt to stop the thread.
My second suggestion is maybe a little uncommon, but I would say let's just ignore the deprecation and call stop(). While not very nice, this is fuzzing-land where we simply need this kind of control, because AFL fuzzing will always find a way to hang Java. As the same doc link suggest, it's probably best to catch ThreadDeath, even if that's not optimal.
There are also suggestions to use the new JDK 9 spinWait(), but I'm not sure that method would help.

I've added some limited support for the -t option when using jqf-afl-fuzz.  The only snag is that some timeouts may get misclassfied as "crashes" instead of "hangs". This is because I'm trying to stick to using stock AFL instead of having a custom AFL for JQF, which means I have to fool AFL into believing that it has killed the target program on timeout by mocking signals. In the worst case, the signal will appear before AFL's timer goes off and it will therefore be classified as a crash instead of a hang.
Note: You will have to re-run the make command in addition to mvn package (or simply setup.sh which does both) because the AFL proxy has also been modified.
The default value for timeout is 10000 (milliseconds) to prevent spurious timeouts, but I've tried fuzzing Tika with 5000 on my machine and it was fine.
The implementation does not use either Thread.stop() nor Thread.interrupt(). Instead, since we are instrumenting program branches for fuzzing, we can simply monitor how much time has elapsed after every few thousand branches (without incurring too much overhead), and then throw a run-time exception if the elapsed time is more than the set timeout. This idea was suggested by Koushik Sen.
This trick means that JQF can detect hangs due to infinite loops fairly efficiently, but not hangs due to deadlocks where no branches are being executed. For now, I will say this is sufficient as most fuzzing targets are not I/O intensive. In the future, we can possible interrupt deadlocks using Thread.interrupt().

This is still not working as expected. Although the first hang gets detected, throwing a TimeoutException from within an instrumentation handler seems to corrupt the instrumentation tracking itself and subsequent hangs do not get detected.
I'm still working on it... the issue remains open.

@floyd-fuh Don't worry about that. We always give AFL a 50ms timeout regardless of what the actual timeout is. This is so that AFL sets its timeout flag for every single input. In practice, AFL only marks a run as a "hang" if the timeout flag is set AND the return value of waitpid() on the forked fuzz target indicates SIGKILL. In JQF, we always let AFL's timeout flag fire (in 50ms) and handle the real timing measurements in Java. When a TimeoutException is thrown, we lie to AFL and tell it that the process ended with SIGKILL.

Ah, perfect, thanks for the pointers! I think I just thought it wouldn't work correctly, but everything is fine.
Btw. I ran into new problems with the timeouts. JQF classified the Java bug found in https://github.com/rohanpadhye/jqf/issues/27 as a crash. As you described above not what you intended, but perfectly fine in my use case. So a miss-classification as crash or hang is fine for me. However, finding a similar bug in a Java library JQF actually got stuck. I think the reason is that the endless loop is in a part of the code that was not instrumented, so JQF won't be able to measure time and throw a run-time exception after the elapsed time.
So at the moment I have to recheck on my fuzzers from time to time. Today they were running for 5 days, but got stuck after the first 7 hours. I wasn't too unhappy because that meant a new finding, but it's obviously not optimal having to check on them and restart fuzzers often.
So I guess a Thread.interrupt solution would be really helpful.
And sorry for asking for so many features, I kind of feel bad, but JQF is really improving a lot and I clearly prefer it over Kelinci and java-afl at the moment. Keep up the good work 👍

Timeouts are working fine in AFL now.  Please re-open if issues persist.

I updated the version of JQF I'm using to the latest github version and started fuzzing Apache Tika again. However, as you said AFL timeout should work fine now, I ran JQF with lower timeouts, such as 1000 or even 300. It runs fine and also the execs/s seem to show that it is going faster. However, stock AFL would usually put a lot of files in the "hangs" directory and would very quickly find hangs. However, JQF never puts any file in the "hangs" directory for me now, which is very weird with these timeout settings.
So when I run with -t 300 I expect every file that takes more than 300ms of processing to be put in the hangs directory. This doesn't seem to be the case. So my guess is at the moment JQF is not able to find endless loops as it doesn't fill the hangs directory. Can you confirm this?
Can you elaborate a little on how the timeout setting is applied?

Hi Floyd,
I think this could be because of Tika's multi-threaded parsers. The last commit changed the behavior to throw TimeoutException(s) only for the first branch-event after the time limit instead of for every event after the time limit, since in some other benchmarks we were seeing timeouts being continuously re-thrown from finally{} blocks that surrounded the (possibly infinite) loops. As much as possible, we would like to execute the finally blocks even if the enclosed loop was infinite.
However, this means that if the timeout occurs in not-the-main-thread, it may not get reported. I can think I can attempt a fix that reports a timeout in the first branch-event per-thread after the time limit is reached.

Pushed an attempted fix. Please let me know if this changes anything. If you still do not see any timeouts (or misclassified crashes), I may have to debug specifically with Tika.

Perfect! Looks much better now... At least I get hangs now, obviously I cannot be sure if it is really detecting all of them :)
Thanks a lot for the fix

Hi Rohan,
Is there any similar solution as the described here for AFL (-t) in zest fuzzing?
I am currently using fuzzing with Zest driver and after some iterations it hangs for long time.
Adding a timeout value to the test method i.e.:
@Fuzz     @Test(timeout = 1000)     public void testNoFilter(...)
will cause an exception inside of JUnitQuickcheck:
Method testNoFilter should have no parameters at com.pholser.junit.quickcheck.runner.JUnitQuickcheck.validateTestMethods(JUnitQuickcheck.java:82).

Hi Rohan,
Just to keep you updated I have tried to define the timeout with a global Rule and it seems to not hang anymore i.e.
```
@Rule
public Timeout globalTimeout = Timeout.millis(500);

But I have then noticed that the total and valid coverage are always at `0 branches (0% of map)` so I guess this timeout rule is breaking the coverage calculation... so it's not a solution or a complete solution.

Hi @Ahmedfir,
In Zest, you can set timeout for individual runs via the system property jqf.ei.TIMEOUT. Perhaps I should add a -t option to the jqf-zest launcher as well to make this easier. It used to be experimental but I believe that it should work just as well as in AFL.

Reopening as jqf-zest needs a -t option.

thank you for the quick answer. It's a nice idea, I think. I didn't use the CLI before. I have only run zest with the maven plugin so-far but I will try CLI with the jqf.ei.TIMEOUT and keep you updated. Thanks again.

It should also work with the maven plugin. The maven plugin runs the same
code. Simply give - Djqf.ei.timeout when running mvn jqf:fuzz.
The maven plugin is the preferred way of running Zest. The command line
script is mainly for trying out experimental features and/or fuzzing the
JDK classes.
On Mon, Sep 23, 2019, 10:33 AM Ahmed Khanfir<EMAIL_ADDRESS>wrote:

thank you for the quick answer. It's a nice idea, I think. I didn't use
the CLI before. I have only run zest with the maven plugin so-far but I
will try CLI with the jqf.ei.TIMEOUT and keep you updated. Thanks again.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/rohanpadhye/jqf/issues/26?email_source=notifications&email_token=ABAJQMUY54Y2LUEWEBY3UWTQLD4V3A5CNFSM4E6Y73Q2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7LUWIQ#issuecomment-534203170,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABAJQMRAKONHQPOI2BLXXYLQLD4V3ANCNFSM4E6Y73QQ
.

Hi @rohanpadhye,
I have tried the maven parameter and it doesn't seem to work for me i.e.
-Djqf.ei.timeout=500

@Ahmedfir Do you know what is causing the hangs? In both the AFL and Zest guidances, we check the timeout during each code coverage event. This means that we can detect timeouts due to potentially infinite loops in instrumented app classes. If the app is timing out because it is waiting for I/O or if it is looping in an uninstrumented class, then JQF cannot detect it. In Java, it is not safe to stop a thread while it is waiting on I/O or compute. Since all of this is essentially in-process fuzzing, we cannot send SIGKILL either.
Can you please confirm the source of your app hangs? Maybe try repro with the last generated input (I believe Zest stores the last input in a file called .cur_input in the results directory).

Hi @rohanpadhye, indeed it is timing out because of a thread running further inside of the test. thank you anyway, I will try to avoid such uses in the future. Cheers

I've added options for both jqf-zest and the maven plugin. For some benchmarks, very low timeouts are still resulting in unexpected exceptions in the instrumentation, which still needs looking into. Let me know if you also encounter such issues.

thank you very much! I am simply trying to not using any extra thread in the test execution.
Nevertheless, I am currently having troubles with the tracing and the coverage calculation (this has nothing to do with the current issue; timeout...).
Basically, I am always getting  0 branches (0.00% of map)
and I suspect it's caused by  a static field used (and modified) in both the test and the generator. But I don't think it is the real cause as I had it working before with static fields.
Or may because I a have a very long executing @BeforeClass method. So the ThreadTracer (the instrumentation for the coverage calculation) traces the @BeforeClass instead of the test method... But I don't think it's really the case because it's getting the test method as entry point.
Did you get such issues before?

Perhaps the test is not running in the main thread? Zest currently only traces the main thread.

It is in the main thread. but now I am doubting about something else. The test is executing code in a jar file that might be invisible to the tracer... hmm I will dive further in this direction and let you know. :) thank you as always for your quick answers!

More undocumented features that may be useful: try setting -Djanala.verbose=true before fuzzing to get a log of all classes that get instrumented (or not). Maybe this needs to be made default when -v option is given.

Thanks, that' great to know!
I got the tracing and the code coverage to work when downgrading my maven configuration target and source to java 9 instead of 12. I don't know exactly what was the issue with java 12... (it can be an issue related to my local setup :p )
PS: I got the coverage working with jdk 12 for simpler examples.
PS: May be JQF doesn't handle well jdk 12 and I've missed this point in the documentation. Else May be it would be nice to mention that in the documentation...
Please don't hesitate to contact me if you want to investigate this mysterious issue further. :)

I was just wondering why Zest has no "hangs" directory in its output directory? Or is it just by default not using any timeout configuration?
I think for Java security analysis it is as important to get all the hangs as the failures, as infinite loops might be really bad.

I believe that timeouts currently get saved in the failures directory
itself, marked with TineoutException in the logs.
I'm open to the idea of adding another "timeouts" directory to the results,
if you think this is an important distinction.
On Wed, Oct 2, 2019, 11:04 PM floyd<EMAIL_ADDRESS>wrote:

I was just wondering why Zest has no "hangs" directory in its output
directory? Or is it just by default not using any timeout configuration?
I think for Java security analysis it is as important to get all the hangs
as the failures, as infinite loops might be really bad.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/rohanpadhye/jqf/issues/26?email_source=notifications&email_token=ABAJQMREYWBUNKEIABEFATTQMWDPLA5CNFSM4E6Y73Q2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAHDFMY#issuecomment-537801395,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABAJQMW4CJTCQQAOX3TCYITQMWDPLANCNFSM4E6Y73QQ
.

Ah, no, that's perfectly fine then. I was just wondering. I just haven't seen any being stored in the failures directory.

so-far I am not seeing any timeout scenario happening.
I have also tried changing (locally) the timeout implementation by passing the timeout directly to the org.junit.runners.BlockJUnit4ClassRunner#methodBlock in the edu.berkeley.cs.jqf.fuzz.junit.TrialRunner by calling the org.junit.runners.BlockJUnit4ClassRunner#withPotentialTimeout method. This would normally reproduce the same behaviour as adding a timeout annotation to a normal junit test method.
Even this solution didn't lead the tests to timeout (knowing that the tests are in the same thread...).
So I think it would be fair to close this ticket with the status "won't fix".
Thank you again for your time. :)

Thanks for the update!


Inference is broken on intersection type utilizing polymorphic this

Bug Report

🔎 Search Terms

inference, intersection, polymorphic this, deferred index access
🕗 Version & Regression Information

This changed between versions 5.1.0-dev.20230313 and 5.1.0-dev.20230315

⏯ Playground Link

Playground link with relevant code
💻 Code

export interface TypeLambda {
  readonly A: unknown
}

export interface TypeClass<F extends TypeLambda> {
  readonly _F: F
}

export type Apply<F extends TypeLambda, A> = F extends { readonly type: unknown }
  ? (F & { readonly A: A })['type']
  : { readonly F: F, readonly A: A }

export interface T<A> {
  value: A
}

export interface TTypeLambda extends TypeLambda {
  readonly type: T<this["A"]>
}

export declare const map: <F extends TypeLambda>(F: TypeClass<F>) => <A, B>(a: Apply<F, A>, f: (a: A) => B) => Apply<F, B>

declare const typeClass: TypeClass<TTypeLambda>

declare const a: T<number>

map(typeClass)
// TS 5.0.4:              <A, B>(a: T<A>, f: (a: A) => B) => T<B>
// TS 5.1.0-dev.20230422: <A, B>(a: (TTypeLambda & { readonly A: A; })["type"], f: (a: A) => B) => (TTypeLambda & { readonly A: B; })["type"]

map(typeClass)(a, (_) => _)
                // ^ TS 5.0.4 infers number
                //   TS 5.1.0-dev.20230422 infers unknown

🙁 Actual behavior
Each version including and after 5.1.0-dev.20230315 defers the index access in the Apply type and doesn't allow inference to continue.
🙂 Expected behavior
Inference should continue to work in this case.
After a bit of research, I think that #53098 is the PR that introduced this change.

This currently breaks the encoding used in @effect/* for HKTs and probably a few other libraries that rely on the same pattern (hotscript, storybook, come to mind)

Same here, zodios is using this pattern to enable policy based interfaces.

Seems like those packages should be added to the RWC test suite maybe, or snippets of them added to the main compiler tests

Feel free to send a PR: https://github.com/microsoft/TypeScript-error-deltas
See userTests.

I can confirm that the change in behavior is caused by #53098. We definitely want the deferred resolution implemented by that PR. However, it means that types like (TTypeLambda & { readonly A: A; })["type"] aren't resolved until a non-generic type is substituted for A through instantiation and, unfortunately, it also means that no inferences are made for A in the map(typeClass) invocation.
I think we can implement a targeted fix where, when inferring to types like (TTypeLambda & { readonly A: A; })["type"], we force the resolution that previously happened and then infer to the result of that. This means the type of map(typeClass) will still change as in the example above, but inference will infer number in the final example.

I don't think we need to be this dramatic.
I see this as an opportunnity to clarify TS team stance on HKT.
Community started evolving libraries to take advantage of some HKT features.
As a TS team you have a playground feedback on how community took advantage of it.
Based on this feedback, you Can now either :

officially support a new HKT syntax to address current use cases
officially announce HKT to not be supported in either way going forward

Clarification is key for us to know if we can leverage some ts features for our use cases.
Again, thank you @ahejlsberg and all ts team for your wonderfull work

I understand that esthetically T<A> looks nicer than (TTypeLambda & { readonly A: A; })["type"], though the latter really is a more precise representation of what was requested in Apply. Are there other issues that are caused by this change and not fixed by #54112?

We use:
export const f = map(typeClass)

and let it derive the type of f, with this change the typing of f will be impractical to use and understand from the reader's perspective, the problem seems to be related to potential reduction to never that this change aims to preserve, would there be a way of avoiding it in a class of cases where never is known to be impossible? I guess that would require being able to express "TypeLambda is an object without the field A" ? or maybe a way to force it to reduce the type?
In a general sense from a user perspective it isn't necessarily the case that something more precise is always better

Actually I may have found a way:
export interface TypeClass<F> {
  readonly _F: F
}

export declare namespace TypeLambda {
  type _A<T> = "A" extends keyof T ? T["A"] : never
}

interface TypeLambda { type: unknown }

export type Apply<F, A> = F extends TypeLambda
  ? (F & {A: A})["type"]
  : { readonly F: F, readonly A: A }

export interface T<A> {
  value: A
}

export declare const map: <F>(F: TypeClass<F>) => <A, B>(a: Apply<F, A>, f: (a: A) => B) => Apply<F, B>

interface TTypeLambda {
  type: T<TypeLambda._A<this>>
}

declare const typeClass: TypeClass<TTypeLambda>

declare const a: T<number>

// const x: <A, B>(a: T<A>, f: (a: A) => B) => T<B>
const x = map(typeClass)

map(typeClass)(a, (_) => _)

Not sure if this if considered a bug with the new behaviour or this can be used

It turns out there's a simple way to fix this issue and preserve our old behavior. See latest commits in #54112.


Blob support

Is there any way to have an Immutable object with a key referencing a Blob? I understand it might not be possible to make the Blob immutable, in which case can I allow having a mutable value within an immutable object?
Failing case:
import Immutable from 'seamless-immutable';
const blob = new Blob();
const test = Immutable.from({blob});
console.log(blob); // logs Blob {size: 0, type: ""}
console.log(test); // logs Object and when expanded blob key is empty

According to the docs, blobs are immutable.

A Blob object represents a file-like object of immutable, raw data.

This suggests we should be able to treat them as such, just like how we currently treat strings!

hi @rtfeldman
in a project im working on I have an issue in safari. uploads to server are failing because the Blob I reference in my state is being treated as data instead of an object to skip (like File/React/Error).
The reason its a Blob and not a File is that creating a File instance from the binary I get (from a 3rd party API) on Safari fails (its empty) and creating a Blob works.
I think Blob should be treated the same as File, no?
Id be happy to submit a PR that does the same as isFileObject for Blob.


Ability to specify list of task ports

In some environments (like containerized environments without isolated network stacks) it is desirable to decide ahead of time exactly what ports your services are going to use. In Druid the main thing stopping this from being easy is the fact that peons have a startPort that you can set, but then they'll use ports potentially above that, and you don't have a ton of control over how high they'll go.
It would be nice to be able to provide a druid.indexer.runner.ports = [8100, 8101, 8102] config as an alternative to druid.indexer.runner.startPort.

+1.
We have a use case where Druid and all its' deps are on Linux, in Docker containers. But the ingestion service is on Windows (don't ask) and using EventReceiver, so we have to expose peons to the outside. Right now, we're just exposing ports 8100-8200 right away and it's usually enough, but it is not ideal, ofc.
Maybe it is good idea to (also?) give option to supply range, like above, to avoid having to set a possibly quite long config array? Just a thought.

+1.
We have met port conflicts because of running middle manger on Docker containers. If more than two middle manger instance run on a same machine, It may lead to conflict.  The way we solve this problem is similar to your suggestion.


Training with .obj files

Hi ZexinHe, sorry to bother!.
I have a question about training with .obj files. Is this possible? or if you can give a hint on how to.
Thanks in advance!

Hi there,
The input and output of OpenLRM are in the format of images. You may need to render your .obj files to images first. https://github.com/3DTopia/OpenLRM?tab=readme-ov-file#data-preparation

Replace this function in the blender script under scripts
# load the glb model
def load_object(object_path: str) -> None:
    """Loads a glb model into the scene."""
    if object_path.suffix == ".glb":
        bpy.ops.import_scene.gltf(filepath=str(object_path), merge_vertices=True)
    elif object_path.suffix == ".fbx":
        bpy.ops.import_scene.fbx(filepath=str(object_path))
    elif object_path.suffix == ".obj":
        bpy.ops.import_scene.obj(filepath=str(object_path))
    else:
        raise ValueError(f"Unsupported file type: {object_path}")


add delete option in product-type.

added an option in product type so that we can also delete it. And also beutify the '/product_type/views.py' file according to flake8.

[x] Your code is flake8 compliant (DefectDojo's code isn't currently flake8 compliant, but we're trying to correct that.)
[ ] If this is a new feature and not a bug fix, you've included the proper documentation in the ReadTheDocs documentation folder. https://github.com/DefectDojo/Documentation/tree/master/docs or provide feature documentation in the PR.
[ ] Add applicable tests to the unit tests.

hello @devGregA  Can you please review this now . Let me know if anything have to change. :smiley:

thanks. @ppiotr3k :smiley:

hello, @devGregA  Can you please review this!

I was about to add this feature myself, until I found this PR. So good news, and thanks for the PR.
I do think the PR stil contains too many changes that are just formatting changes and not related to the added functionality. It does look OK but it requires us to mix and match different lines to see what/if there are changes. It would easier to review if the PR contained just the new stuff.

hello, @valentijnscholten thanks for reviewing. Can you suggest to me what can I do now so that it would be easier to review in this PR?

Make sure views.py only contains logic changes, so remove the changes that only are modifying whitespace or code formatting. (not sure if the flake8 will complain about that)

thanks! done.

Looks good, just checking with @aaronweaver what the policy around the access linting suggestion is?

@valentijnscholten It's mostly looking for labels etc to assist in readability with screen readers. I just added it so I'll check and see what other items it checks.

@aaronweaver was just asking to see if all accesslint issues need to be resolved berfore we can approve PRs

@valentijnscholten Yes, let's start doing it we have at least one user who will benefit from small changes like this.

ok, good to know. We'll wait then for @dr3dd589 to make the suggested changes.

I would like to approve this, but Github doesn't offer me the "add your review" option. Probably because I already approved it earlier? However when I click "Show all reviewers", it only shows the review of @adracea ...

@valentijnscholten weird. Let me see if I can figure it out

Very strange. It shows you approving in the top right of the page.

It only shows a green tick at Andrea's name, mine is black. Maybe just merge the pr and forget about it 🙂

On hovering tick sign it shows:
only reviews by reviewers with write access count towards mergeability

When I hover, it says "valentijnscholten approved these changes". Could it be a bug in github because the PR was created before I got write access?

No , just hover on bottom tick where you approved this, not top right corner tick.

Ah yes, still doesn't make sense because I have write access. Must be a github hickup, let's merge :D

Merged :)


Update parsing entities, remove TI lookup

Parsing entities was using the old trigger
TI is redundant with the TI module
Update operation nameon MCAS module
Fixes #156

@briandelmsft actually https://github.com/briandelmsft/SentinelAutomationModules/pull/155/commits/31b566b0f017ecbaf5695cac9790aaf3e13f5ebb should not be merged. It will be superseeded by the new version of the module.

Commit 2e7069f removes the TI related inputs/outputs from the STAT connector


fix: forced node pool recreation

Fixing https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/issues/1823

I thought I would exhaust all options before committing to a fork. I've tried

Removing the gcfs_config from the remote state and hacking it
Removing all references of gcfs_config from the terraform, no luck
Removing gcfs_config from the node_pool resource, and running a terragrunt state rm and terragrunt import but no luck either

I'm pretty sure what's happening is that it's doing an in memory refresh of the state during a plan, and the API call back contains gcfs_config with the enabled: false field set and it's using that when doing the diff with the local changes. I think that because a couple of these fields don't exist in terraform at all, e.g. fast_socket.
"node_config": [
    {
    "advanced_machine_features": [],
    "boot_disk_kms_key": "",
    "confidential_nodes": [],
    "disk_size_gb": 100,
    "disk_type": "pd-standard",
    "effective_taints": [],
    "enable_confidential_storage": false,
    "ephemeral_storage_config": [],
    "ephemeral_storage_local_ssd_config": [],
    "fast_socket": [],
    "gcfs_config": [
        {
        "enabled": false
        }
    ]

I was hoping I could totally remove all fields from the terraform so i could just apply it using something that ignored it, but no sadly not. Fork it is...


Add Julia v1.11 to CI and buildkite config

Since the Julia v1.11 has released, we should update the CI config and buildkite config

What about the other CIs? I don't remember, like the benchmark tracking and others?

What about the other CIs? I don't remember, like the benchmark tracking and others?

Those always use version = 1, so should be okay.


How to connect coc-tsserver to external tsserver thats already running?

I currently use docker compose for development accross many projects. Ideally I would like to connect coc.vim to an external instance of tsserver running where I plan to run the tsserver using docker and exposing a port on my host machine
I was able to do this with solargraph by specifying the transport method is external and specifying the port number. See the config example below:
{
  "solargraph.transport": "external",
  "solargraph.externalServer": {
    "host": "localhost",
    "port": 7658
  },
}

If I can do something similar for tsserver that would be awesome.

tsserver didn't support this.


Generic LLM wrapper to support chat model interface with configurable chat prompt format

Update 2023-09-08
This PR now supports further models in addition to Lllama-2 chat models. See this comment for further details. The title of this PR has been updated accordingly.
Original PR description
This PR adds a generic Llama2Chat model, a wrapper for LLMs able to serve Llama-2 chat models (like LlamaCPP, HuggingFaceTextGenInference, ...). It implements BaseChatModel, converts a list of chat messages into the required Llama-2 chat prompt format and forwards the formatted prompt as str to the wrapped LLM. Usage example:
# uses a locally hosted Llama2 chat model
llm = HuggingFaceTextGenInference(
    inference_server_url="http://<IP_ADDRESS>:8080/",
    max_new_tokens=512,
    top_k=50,
    temperature=0.1,
    repetition_penalty=1.03,
)

# Wrap llm to support Llama2 chat prompt format.
# Resulting model is a chat model
model = Llama2Chat(llm=llm)

messages = [
    SystemMessage(content="You are a helpful assistant."),
    MessagesPlaceholder(variable_name="chat_history"),
    HumanMessagePromptTemplate.from_template("{text}"),
]

prompt = ChatPromptTemplate.from_messages(messages)
memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)
chain = LLMChain(llm=model, prompt=prompt, memory=memory)

# use chat model in a conversation
# ...

Also part of this PR are tests and a demo notebook.

Tag maintainer: @hwchase17
Twitter handle: @mrt1nz

Not included yet is support for function calls (i.e. FunctionMessage). I plan to add this in another PR.

@baskaryan @rlancemartin First of all, apologies for the extra round caused by linting errors from my first commit which are now resolved by my second commit.
This was related to some issues I had running all make target locally, for which a fix is now available in #8344. I'm still experiencing the issues mentioned in #6182 though but was able to workaround by using /usr/bin/make directly.

@baskaryan @rlancemartin First of all, apologies for the extra round caused by linting errors from my first commit which are now resolved by my second commit.
This was related to some issues I had running all make targets locally, for which a fix is now available in #8344. I'm still experiencing the issues mentioned in #6182 though but was able to workaround by using /usr/bin/make directly.

Thanks for adding!
Re function calling, have you seen this and do you have a PR in flight to support it? Would be great!

Thanks for reviewing my PR @rlancemartin! I haven't seen the grammar-based sampling PR, thanks for the pointer. I'll start working on it later this week and submit another PR.

@rlancemartin I just added two more commits that include an example how to use the Llama2Chat wrapper with a local LlamaCPP LLM and further tests incl. failure cases. I also tried to improve the documentation in the notebook.

@rlancemartin @baskaryan is there anything else you think I should add/improve to get this PR merged?

@krasserm The llm created by HuggingFaceTextGenInference doesn't support async operation, so it fails in async mode. Do you notice that?

@yaoaifiling async support for HuggingFaceTextGenInference was recently added. Are you using an older LangChain version?

@yaoaifiling async support for HuggingFaceTextGenInference was recently added. Are you using an older LangChain version?

Thanks! Yes, using an old one, need to update my langchain version.

@rlancemartin @baskaryan is there anything else you think I should add/improve to get this PR merged?

apologies for the delay, will take another look shortly!

Thank you @krasserm for doing this!
By the way, I tried it out and wanted to see how good it extends. By default it doesn't work with e.g. Vicuna, mostly because the prompt templates are different. But based on https://github.com/lm-sys/FastChat/blob/main/fastchat/conversation.py#L336 I was able to make it work w/ Vicuna with very simple modifications to _to_llama2_chat_prompt - to make the template look like
{SYS} USER: {message[2n]} ASSISTANT: {message[2n+1]} </s>USER: {message[2(n+1)]} ASSISTANT: {message[2(n+1)+1]} </s>

So, this looks promising as a chat wrapper for some major local llama-based models, if templating mechanism to be modified. I also have AutoGPTQ pipeline added to LangChain locally (will probably fire a PR soon), which allows me to run quantized models easily.
How do you think this should go? This generalization is probably out of scope of this PR? If so, then we can potentially duplicate the Llama2Chat into something like LlamaLocalChat or something with generalized templating.

Good thoughts @avoroshilov, thank you! I just updated this PR to make the chat wrapper generic.  The new base class ChatWrapper can now be configured to support custom chat prompt formats e.g.
class Llama2Chat(ChatWrapper):
    sys_beg: str = "<s>[INST] <<SYS>>\n"
    sys_end: str = "\n<</SYS>>\n\n"
    ai_n_beg: str = " "
    ai_n_end: str = " </s>"
    usr_n_beg: str = "<s>[INST] "
    usr_n_end: str = " [/INST]"
    usr_0_beg: str = ""
    usr_0_end: str = " [/INST]"

class Llama2Instruct(ChatWrapper):
    sys_beg: str = "### System:\n"
    sys_end: str = "\n\n"
    ai_n_beg: str = "### Assistant:\n"
    ai_n_end: str = "\n\n"
    usr_n_beg: str = "### User:\n"
    usr_n_end: str = "\n\n"

class Vicuna(ChatWrapper):
    sys_beg: str = ""
    sys_end: str = " "
    ai_n_beg: str = "ASSISTANT: "
    ai_n_end: str = " </s>"
    usr_n_beg: str = "USER: "
    usr_n_end: str = " "

Lllama2Chat is a refactoring of the original implementation. Llama2Instruct is a wrapper for llama-2-*-instruct-v2 models (and also several other Llama-2 derivatives like StableBeluga-2) that require a prompt format like
### System:
{System}

### User:
{User}

### Assistant:
{Assistant}

Vicuna is for the prompt template you posted i.e.
{SYS} USER: {message[2n]} ASSISTANT: {message[2n+1]} </s>USER: {message[2(n+1)]} ASSISTANT: {message[2(n+1)+1]} </s>

I hope this fits your use cases!

@baskaryan @rlancemartin it would be great if you could take another look at this PR. Based on @avoroshilov's thoughts I made this a generic LLM wrapper that now supports a lot more models than just Llama-2 chat models. Models not already covered can now be easily supported by extending ChatWrapper base class. See this comment details.

Hi team, any update on this PR? I don't see llama-2-based chat model support in LangChain yet. So being able to merge this would be really awesome.

@krasserm Hi , could you, please, resolve the merging issues and address the last comments (if needed)? After that, ping me and I push this PR for the review. Thanks!

@leo-gan thanks for your support! I just rebased on master and made some changes to the notebook (support for latest llama-cpp-python version, ...). I also squashed all commits from this PR.

@baskaryan Please review this PR. TNX

@leo-gan any updates on this? Would appreciate feedback on how you'd like to proceed with this PR.

@efriis Please, take a look.

Hey folks! Appreciate your patience. Given this is effectively a model-specific prompt template, I would prefer to have this as a reference prompt at smith.langchain.com/hub , or as a LangChain template for working with open source models instead of a "model" as it's implemented here.
If nobody else picks it up, I can spend some time putting together a little template with alternatives showing how to format this, so people can get started with these llms quicker with that, without merging these prompts into the models codebase.

Hey folks! Appreciate your patience. Given this is effectively a model-specific prompt template, I would prefer to have this as a reference prompt at smith.langchain.com/hub , or as a LangChain template for working with open source models instead of a "model" as it's implemented here.
If nobody else picks it up, I can spend some time putting together a little template with alternatives showing how to format this, so people can get started with these llms quicker with that, without merging these prompts into the models codebase.

Hi @efriis
I guess this PR could have changed in a way to not call on each LLM model, but any LLM model that requires a specific start/end for system, user, ai, etc messages. However, if we are not going to merge this, I am interested in LangChain template for working with open source models solution which doesn't require downloading something extra from a Hub or installing another dependency just to get something like Llama-2 work.

I decided for this design because I see start/end tokens in chat prompts as an implementation detail of a model. This is something a user shouldn't need to care about when designing an application-specific chat prompt template like the one used in the initial example e.g.
messages = [
    SystemMessage(content="You are a helpful assistant."),
    MessagesPlaceholder(variable_name="chat_history"),
    HumanMessagePromptTemplate.from_template("{text}"),
]

# independent of model-specific chat message start/end tokens
prompt_template = ChatPromptTemplate.from_messages(messages)

Start/end tokens for messages of a chat model are an implementation detail in the same way as a tokenizer is a model-specific implementation detail. I think that's the reason why Hugging Face decided to link chat templates to tokenizers. Their chat templates too render model-independent chat messages into prompts with model-specific start/end tokens (like the wrappers of this PR) and are reusable across tokenizers.
You can also see the wrapper(s) from this PR as a layer in a chat protocol, translating messages of a conversation into a lower-level, model-specific prompt format, freeing the user from dealing with these details. And these wrappers (or protocol layers) are reusable too. For example, you can reuse the Llama2Chat wrapper with all LLMs that require the Llama-2 chat prompt (and many of them do, not only vanilla Llama-2 chat models).
Even if you do not want to have concrete subclasses, like Llama2Chat, part of this PR, I think the ChatWrapper base class is still a useful addition to the models codebase as it provides the generic functionality for implementing a uniform chat model interface for many LangChain LLMs. For example, many models require the ChatML prompt format which can be implemented with ChatWrapper easily like:
class ChatML(ChatWrapper):
    sys_beg: str = "<|im_start|>system\n"
    sys_end: str = "\n<|im_end|>"
    ai_n_beg: str = "<|im_start|>assistant\n"
    ai_n_end: str = "\n<|im_end|>"
    usr_n_beg: str = "<|im_start|>user\n"
    usr_n_end: str = "\n<|im_end|>"

Their wide applicability is one of the main reasons why I thought concrete ChatWrapper implementations should be part of this PR.

@krasserm thoughts on merging this into experimental initially? Code can stay almost the same - all the dependencies would still come from langchain, but the new stuff would come from langchain_experimental while we ask people to try it out and let us know their thoughts, and how it plays with the different models.

@efriis sounds good to me. I'll update the PR later this week.

Sounds good. Thanks!
Another (better imo) implementation is as some kind of function that formats ChatPrompts into the format expected by these LLMs, and an output parser that parses the output into a message (or messages).
I would imagine a runnable like
chain = ChatPromptTemplate.from_message([...]) | new_prompt_formatter | HuggingFaceTextGenInterface(...) | new_llama_output_parser

Sounds good. Thanks!
Another (better imo) implementation is as some kind of function that formats ChatPrompts into the format expected by these LLMs, and an output parser that parses the output into a message (or messages).
I would imagine a runnable like
chain = ChatPromptTemplate.from_message([...]) | new_prompt_formatter | HuggingFaceTextGenInterface(...) | new_llama_output_parser

It would be great if that ChatPromptTemplate can have a format that can be applied to every chain, agent, etc. that uses any ChatPrompt messages. This does override the ChatPromptValue for everything else https://github.com/langchain-ai/langchain/issues/9917

Sounds good. Thanks!
Another (better imo) implementation is as some kind of function that formats ChatPrompts into the format expected by these LLMs, and an output parser that parses the output into a message (or messages).
I would imagine a runnable like
chain = ChatPromptTemplate.from_message([...]) | new_prompt_formatter | HuggingFaceTextGenInterface(...) | new_llama_output_parser

So you're basically creating a chain that allows using an LLM as a chat model without using an implementation of BaseChatModel. What is the purpose of the BaseChatModel asbtraction then?
From my perspective, BaseChatModel implementations should also be responsible for hiding details like serializing chat messages into low-level prompt strings. Users of BaseChatModel implementations don't (and shouldn't) need to care about these details.
For example, when using ChatOpenAI, details like separating chat messages within a lower-level prompt string are also hidden from the user (it is even hidden behind the OpenAI API but that's an implementation detail too).
So why shouldn't LangChain try to follow these design principles of chat model interfaces consistently i.e. encapsulating details like chat messages separators, tokenization, ... etc. behind chat model interfaces instead of sometimes exposing it to users (like in your proposal) and sometimes not?
Having a common chat model interface with clear semantics (which includes dealing with low-level chat message separators) would simplifiy application development with LangChain a lot.

I just moved everything from langchain to experimental and updated the demo notebook accordingly. I also added a pytest-asyncio dependency to pyproject.toml needed for executing async tests. An updated poetry.lock is included.

Appreciate it!
Re: the previous point about composing the LLM rather than wrapping it in the BaseChatModel - I think this is a discussion of simplicity vs ease of use.
ChatOpenAI makes sense as a purpose-built chat interface because it consistently returns the correct format. With many of these self-hosted LLMs, getting that chat model kind of consistency is difficult, so having simpler abstractions that have to be composed together makes more sense to me.
That being said, these kinds of experiments are what experimental is for! Thanks for moving it

Landed! Thanks @krasserm and appreciate your patience on this one

Thanks @efriis for discussing and supporting this PR


v0.3.13 TypeError: Cannot assign to read only property

After updating to from 0.3.12 to 0.3.13 I am getting the following error:
  error: [TypeError: Cannot assign to read only property '/path/1/' of [object Object]] }
    at Logger.error (webpack:///./src/helpers/logger.js:102:15)
    at process.<anonymous> (webpack:///./src/server.js:69:16)
    at emitTwo (events.js:92:20)
    at process.emit (events.js:172:7)
    at process.emit (/Users/chris/github/app/node_modules/source-map-support/source-map-support.js:419:21)
    at /Users/chris/github/app/node_modules/core-js/modules/es6.promise.js:113:19
    at perform (/Users/chris/github/app/node_modules/core-js/modules/es6.promise.js:59:5)
    at Immediate._onImmediate (/Users/chris/github/app/node_modules/core-js/modules/es6.promise.js:111:16)
    at processImmediate [as _immediateCallback] (timers.js:383:17)

Is this expected in the new release? (pointing out an error in my code)

v0.3.13 enfoces immutability, maybe you are tryng to modify a frozen object? The debugger should help in these cases

This was the issue I was running into. Thank you


Does this support heterogeneous lookup?

Does this unordered_map variant support heterogeneous lookup using the is_transparent flag?
If so, maybe add it to the docs?

Yes it does, no docs yet though. Search in the unit test and you'll find an example

Until there are docs, if anyone is interested the unit tests are here:  https://github.com/martinus/unordered_dense/blob/main/test/unit/transparent.cpp
Thanks @martinus for the speedy answer!


Rack v3: Streams

What
This PR is a prototype that updates twirp-ruby to work with Rack v3 & streamed requests & responses. I've introduces a Twirp::Stream class that wraps the Rack stream to add input & output encoding.
Issues
This list probably isn't complete but theres a few known issues with my changes:

Along the way I removed the @on_error handlers to make the refactor simpler. They need to be reimplemented.
self.classraise_exceptions` also needs to be reimplemented.
The API of the Handler classes now take a Twirp::Stream & are expected to call #read & #write.

You can still return the result from the Handler & the stream will automatically call #write.
For most cases where input isn't expected to be a stream it would make sense to pre-read from the stream & pass input like the old API. Perhaps the stream input API could be handled by a special method e.g. (env[:ruby_stream_method], instead of env[:ruby_method])

protoc-gen-twirp_ruby could be updated to generate some different code when the proto file indicates the input can be streamed.

This was just a prototype


Whose view is not in the window hierarchy!

when i try to make
element.on("webkitbeginfullscreen", function() {
console.log("webkitbeginfullscreen");
screen.lockOrientation('landscape-primary');
});
i`m getting
Warning: Attempt to present <ForcedViewController: 0x16daaaa0> on <MainViewController: 0x155c2460> whose view is not in the window hierarchy!
any ideas?

Hi @morkov4o
This plugin is based on an early version of the screen orientation api and does not support the fullscreen events.  What are you trying to achieve in your app? In cordova you should not need any of the fullscreen functions, the app is already fullscreen and if you want to hide the status bar you can use the statusbar plugin.

Hi,
thanks for the quick response. We have HTML5 video player and I want to force the full screen video to play in landscape mode. Thats why im using the screen orientation api.

Hi @morkov4o
If you provide me with a working/failing test case then I can look into it further for you, iOS and screen orientation is a bit of a minefield.  It is achieved by presenting and removing a view over the top of the cordova view and has been broken in the past by ios updates.


remove pnpm-workspace.yaml

Description
remove pnpm-workspace.yaml file, because most of the time, the vscode extension will not need to workspace such a directory structure

Linked Issues
Not yet.
Additional context
Not yet.

Thanks but no. Feel free to remove it on your own fork.


Implement CPFValueObject as util Value Object

CPF stands for Cadastro de Pessoas Físicas (Natural Persons Register). Native Brazilians can request a CPF at any time in their lives - parents will often apply on behalf of their children when they are newborns, as CPFs are vital to living in Brazil and it’s best to obtain one sooner rather than later.
A CPF is 11 digits long, comprised of nine base digits, and two digits at the end that are the result of an arithmetic operation on the first nine numbers, meaning any typing mistakes will lead to an invalid number. Here is an example of a CPF: 231.002.999-00.

Validate digits
Reference Doc
Possible Province
Example CPF nº 000.000.006-00
0 - Rio Grande do Sul
1 - Distrito Federal, Goiás, Mato Grosso do Sul e Tocantins
2 - Pará, Amazonas, Acre, Amapá, Rondônia e Roraima
3 - Ceará, Maranhão e Piauí
4 - Pernambuco, Rio Grande do Norte, Paraíba e Alagoas
5 - Bahia e Sergipe
6 - Minas Gerais
7 - Rio de Janeiro e Espírito Santo
8 - São Paulo
9 - Paraná e Santa Catarina

Regex to validate CPF
/^([0-9]{3})[\.]((?!\1)[0-9]{3})[\.]([0-9]{3})[-]([0-9]{2})$|^[0-9]{11}$/

Remove special chars
cpf.replace(/[\.]|[-]/g,'');


Add tests for setting a channel to lame on an invalid default service config

Add tests for setting a channel to lame on an invalid default service config.
This results in all RPCs on that channel to fail.
Note that the TODO mentions that the channel should be set in TRANSIENT_FAILURE. Offline discussion might be needed. #18890 is where the discussion till now has been

@markdroth I'm wondering if I can simply modify SetServceConfigJSON to return a bool, or alternatively, provide some API to check if the channel returned is a lame channel or not.

If the default service config is invalid, I think we want to actually return the parse errors, not just a bool indidcating it was invalid.  Otherwise, the user won't know what to fix.
I think we're going to need to expose some sort of validation function that does the parsing and returns any errors if the parsing fails.  We just need to ensure that the validation function is called after all of the parsers are registered.

Known issues : #18967, #18968, #18892


core: Added comparison in parse_result method

Issue:
Occurring with watsonx implementation.
Steps:
class Joke(BaseModel):
    """Joke to tell user."""

    setup: str = Field(description="question to set up a joke")
    punchline: str = Field(description="answer to resolve the joke")

chat = model.with_structured_output(Joke)

for chunk in chat.stream("Tell me a joke about cats."):
    assert isinstance(chunk, Joke)

Error:
self = PydanticToolsParser(first_tool_only=True, tools=[<class 'langchain_standard_tests.integration_tests.chat_models.Joke'>])
result = [ChatGenerationChunk(message=AIMessageChunk(content='', additional_kwargs={'tool_calls': [{'index': 0, 'id': 'chatcmpl...otal_tokens': 216}, tool_call_chunks=[{'name': None, 'args': '', 'id': None, 'index': 0, 'type': 'tool_call_chunk'}]))]

    def parse_result(self, result: list[Generation], *, partial: bool = False) -> Any:
        """Parse the result of an LLM call to a list of Pydantic objects.

        Args:
            result: The result of the LLM call.
            partial: Whether to parse partial JSON.
                If True, the output will be a JSON object containing
                all the keys that have been returned so far.
                If False, the output will be the full JSON object.
                Default is False.

        Returns:
            The parsed Pydantic objects.

        Raises:
            OutputParserException: If the output is not valid JSON.
        """
        json_results = super().parse_result(result, partial=partial)
        if not json_results:
            return None if self.first_tool_only else []

        json_results = [json_results] if self.first_tool_only else json_results
        name_dict = {tool.__name__: tool for tool in self.tools}
        pydantic_objects = []
        for res in json_results:
            try:
                if not isinstance(res["args"], dict):
                    raise ValueError(
                        f"Tool arguments must be specified as a dict, received: "
                        f"{res['args']}"
                    )
>               pydantic_objects.append(name_dict[res["type"]](**res["args"]))
E               KeyError: ''

Solution:

Added key validation

could you write a unit test for the output parser showing which input you want to parse? It looks like your LLM might be generating a field that doesn't exist on the provided schema.

It has been resolved on watsonx side. Closing


put batch_processing directly in the package

At the same level than the installer.py

fixed in commit 30274ca93a3fb0c6a411701186e9aa71b6e86339


Add other dp events, and fix possible infinite digest loop

Adds the previously unimplemented items from http://eonasdan.github.io/bootstrap-datetimepicker/Events/
Fixes digest loop i mentioned in #26  and by doing so allows for the functionality desired in #26

@CapitanMorgan whoa, why the tests failed so badly ?

When i check out master locally, it was already broken. So I started about trying to fix it.

You are right. So at the time of last commit to master Travis was OK, but actually, now it does not pass.
https://travis-ci.org/atais/angular-eonasdan-datetimepicker/builds/197936266
So I think 1st thing would be to sort this out and later I can think of adding new stuff.

I fixed the tests in #29

So ideally I would like you to create a test scenario describing this infinite loop that would fail on current master.
And from there you could create a fix for it and present the whole as a PR.


[FEATURE] Specify Web Drivers File Name

Describe the feature
Requesting improvement to BrowserDriverFactory.cs to use a new BrowserOptions parameter that specifies the driver executable file name when calling the various DriverService.CreateDefaultService functions.
example:

var edgeService = string.IsNullOrWhiteSpace(options.DriversExecutable) ? EdgeDriverService.CreateDefaultService(options.DriversPath) : EdgeDriverService.CreateDefaultService(options.DriversPath, options.DriversExecutable);`

Explain why this feature is needed
EasyRepro BrowserDriverFactory.cs currently only passes the DriversPath into the Selenium CreateDefaultService function, however CreateDefaultService supports a second string parameter for specifying the driver executable file name.
Azure Pipeline Agent's have built in WebDriver executables that stay in sync with the built in browser, however the webdriver file name cannot be modified from their the defaults (ex: "chromedriver.exe" or "msedgedriver.exe") which do not match the default file names assumed by Selenium (ex: "MicrosoftWebDriver.exe" for Edge).
Additional context
Tests currently throw exception in pipeline if passing exe in path name:

Test method Dynamics365.UIAutomation.Tests.Create.CreateTests threw exception:
OpenQA.Selenium.DriverServiceNotFoundException: The file C:\SeleniumWebDrivers\EdgeWebDriver\msedgedriver.exe\MicrosoftWebDriver.exe does not exist. The driver can be downloaded at http://go.microsoft.com/fwlink/?LinkId=619687

Currently the only way to specify the webdriver is to remove the EasyRepro CreateWebDriver function in tests and rebuild the logic from scratch using Selenium.

@zekeinclusa Do you want to use Edge Chromium? If yes it should be feature to support Edge Chromium because of the current implementation for old Edge (non-Chromium).

@zekeinclusa I just say that ability to specify driver file name is not the right solution.
The current implementation for Edge is expected and will work only with old Edge driver MicrosoftWebDriver.exe .
For using Edge Chromium msedgedriver.exe new method should be implemented in EasyRepro (just changing the file name is not enough, because they are different, diff compatibilities, etc...)
I will try to upload the implementation for Chromium Edge soon.

@Shakevg Sounds good, thanks for looking into this.
FYI, pulling the repo and manually setting the file name using the Edge Chromium driver worked for the tests I've been running, albeit with a few long pauses. Biggest downside is maintaining my own branch and manually rebasing latest changes from GitHub.
There are a few steps early on that stop and wait until DefaultTimeout expires (likely waiting for elements unique to the old Edge), but the tests continue and pass. Tests are performing basic actions like Navigation (open app/subArea), CommandBar click command, entity (set value, select tab, etc.).


tool that interfaces with scheduler for long-running tasks

Motivation
The current implementation of tools works for fast toy calculations, but scientifically relevant calculations in chemistry and materials science often make tradeoffs between compute cost and accuracy that results in calculations that run several hours or days, even on powerful hardware.
In the current implementation, the notebook will be blocked for the time of the calculation the calculation will be killed once the ipython kernel is stopped.
We would therefore like langsim to be able to submit computationally intensive tasks to remote scheduling systems, check the status of these calculations, and retrieve the result once they have completed.
Thoughts
I think this is a tough one to make user friendly, particularly if you think about the original target audience: an experimentalist wanting to run calculations.
Do we ask them to install slurm on their local work station (they may be running Windows)? Do they need to apply for computational time on a HPC resource (and then figure out how to run the simulation code they need there)? I think with such asks we already lose a large fraction of the target audience.
The only feasible way I see for letting someone without HPC expertise run on HPC is either

A) Their computational colleagues configure the setup for them
B) They get some cloud account and let langsim connect to a dedicated cloud service for running DFT calculations with a well-defined API [1]

That said, adding the basic functionality for interacting with schedulers is certainly feasible, if the user can provide all necessary information (credentials for connecting, scheduler type, partitions you have access to, where codes are located, etc.).
There is some light at the end of the tunnel, as also academic HPC centers are moving from giving users SSH access to REST APIs (example), but this process is still underway and to my  knowledge no clear standard has emerged.
Also, none of the APIs I've seen so far offer a mechanism for discovering the simulation codes that are installed and how to module load them... perhaps we could draft a specification for how we would like such an API to look like and then approach HPC centers with this idea.
[1] Or, if that is not available, some HPC cluster template with pre-installed software in standard locations (e.g. there are interesting efforts like the CernVMFS build cache from Compute Canada or also the spack build caches), but that already adds a lot of complexity.

I recommend prefect.io for pythonic way to submit custom and monitor pythonic jobs!
I also support different job management systems and can be orchestrated both locally and on HPC


[SQL][DataFrame] Remove DataFrameApi, ExpressionApi, and GroupedDataFrameApi

They were there mostly for code review and easier check of the API. I don't think they need to be there anymore.

Test build #26626 has started for   PR 4328 at commit e00ffcb.

This patch merges cleanly.

Test build #26634 has started for   PR 4328 at commit 723d600.

This patch merges cleanly.

Test build #26626 has finished for   PR 4328 at commit e00ffcb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26626/
Test PASSed.

Test build #26634 has finished for   PR 4328 at commit 723d600.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26634/
Test PASSed.


Stream can be called only once

Hi, i have a problem, that i can call a function using StreamSink on the rust side as parameter only once in dart.
Example:
pub fn my_method(
    sink: StreamSink<String>) {
   // Call sink.add(..)
}

At the dart side:
Future<void> another_method(ValueSetter<String> callback) {
    // init rustLib
     await for (final event in my_method()) {
      callback(event)
    }
}

I call the another_method in my flutter app.
I works without a problem i can show a progress bar progress by using the callback in my UI.
BUT the desired behavior happens only the first time i call another_method.
If i call another_method a second time the stream is already completed but the rust part is executed successfully.
So i see no progress bar progressing the second time :-)
Is this desired behavior because its a single subscription stream? But the stream is created every time the method is called, or am I wrong?
I've already tried closing the subscription and creating a new one every time another_method is called, but that doesn't seem to help
How can i call this function multiple times by using streams?
Best regards

Hi, could you please provide a minimal reproducible sample?
If you call my_method multi times, each time it should have a separate stream and thus the dart callback be called separately.

Now i know that my assumption was right.  Thank you for that! I will try to create a small repro.

 Feel free to ping me when you have a repro

Here is my repro
flutter_rust_bridge_playground-main.zip
I think i find a solution to the problem. But i think it could still be interessting:
use std::{thread::sleep, time::Duration};

use flutter_rust_bridge::frb;
use once_cell::sync::OnceCell;

use crate::frb_generated::StreamSink;

#[flutter_rust_bridge::frb(sync)] // Synchronous mode for simplicity of the demo
pub fn greet(name: String) -> String {
    format!("Hello, {name}!")
}

#[flutter_rust_bridge::frb(init)]
pub fn init_app() {
    // Default utilities - feel free to customize
    flutter_rust_bridge::setup_default_user_utils();
}

#[frb(ignore)]
fn delegate(word: String, current: i32, max: i32) {
    MEM.get()
        .unwrap()
        .add(Progress { word, current, max })
        .unwrap();
}

static MEM: OnceCell<StreamSink<Progress>> = OnceCell::new();

pub struct Progress {
    pub current: i32,
    pub max: i32,
    pub word: String,
}

pub fn words_oncecell(sink: StreamSink<Progress>) {
    MEM.get_or_init(|| sink);
    inner_other(delegate);
}

pub fn words(sink: StreamSink<Progress>) {
    inner(|word, current, max| sink.add(Progress { word, current, max }).unwrap())
}

fn inner<T: Fn(String, i32, i32)>(callback: T) {
    callback("Hello".to_owned(), 1, 4);
    sleep(Duration::from_secs(3));
    callback("Peter".to_owned(), 2, 4);
    sleep(Duration::from_secs(1));
    callback("Hans".to_owned(), 3, 4);
    sleep(Duration::from_secs(2));
    callback("Robert".to_owned(), 4, 4);
}
fn inner_other(callback: fn(word: String, current: i32, max: i32)) {
    callback("Hello".to_owned(), 1, 4);
    sleep(Duration::from_secs(3));
    callback("Peter".to_owned(), 2, 4);
    sleep(Duration::from_secs(1));
    callback("Hans".to_owned(), 3, 4);
    sleep(Duration::from_secs(2));
    callback("Robert".to_owned(), 4, 4);
}

My original code was using words_oncecell function with oncecell. And this seems caused the error.
Now i'am using the approach used in the wordsfunction and it works as expected!

My original code was using words_oncecell function with oncecell. And this seems caused the error.

Not checked into the details (the minimal sample seems still a bit long and can be reduced if needed), but it seems that "once" cell means it will only initialize once. Thus, the second time you call it, it will not be stored. Thus the stream related things only appear once.
Btw, instead of global variables (e.g. the OnceCell), you can use methods attached to structs.

Close since this seems to be solved, but feel free to reopen if you have any questions!


Add an IRC channel

What do you think makes this necessary? I have no idea how to manage that and I don't think I'd be available often, but if there's a demand from the community I can look into it.


respect DNS servers from /etc/resolv.conf

This unhardcodes the google DNS servers.
The Google DNS servers are fine for 95% of the users, but in my case, because of outbound UDP blocking, Google's DNS is blocked.
Inside a docker-container, /etc/resolv.conf should always point to a correct (and working) DNS server. The nameserver-values are either determined automagically by docker or the sysadmin should have configured dockerd with the correct --dns-switch. If one of these things had been done, specifying the -n-option in ovpn_genconfig is not neccessary anymore if DNS-setup is unconventional.

It would be even more dynamic to pull this info from the runtime environment of the docker container, which makes the most sense if people don't hardcode the resolver value.
In ovpn_run:
addArg "--push" "dhcp-option DNS ${OVPN_DNS}"

The Google DNS servers are fine for 95% of the users,

For some users there may be privacy reasons for not-using public/logging/unencrypted DNS servers

So, what happens if the host has namserver <IP_ADDRESS> in /etc/resolv.conf?  It would seem that this would break.
Also the the test conf options doesn't seem relevant with the awk proposal and nothing currently implements the -n arguments.

docker filter out localhost adresses from the host's/etc/resolv.conf because they're indeed unreachable by containers. If there are no nameserver entry left after filtering, Google DNS nameservers are added to the container's /etc/resolv.conf.

Wow, didn't realize that, that seems pretty handy then.

So, what happens if the host has namserver <IP_ADDRESS> in /etc/resolv.conf? It would seem that this would break.

ovpn_genconfig is always executed inside the docker container, not on the docker host.
As the container is not running a dns-server inside it, having <IP_ADDRESS> as nameserver inside the docker container, would break DNS resolution for the container. So this is an unlikely value, which should get caught.

I have concern with respect to the host's DNS settings changing and the config file being out of date.  It seems that  pieterlange's proposal would solve that, but then we'll have the options from the config file pushed + options from the runtime configuration.

[blah blah blah] So this is unlikely to get <IP_ADDRESS>.

 On all my docker containers I've got the host <IP_ADDRESS>. 🤦.
This is completely unexpected. I had in mind, my containers are configured to use the outer server and not localhost. I also have switched just recently from docker 1.12 to 17.04, and I can't test if this update or something else changed it.
This PR is just invalid. Sorry for the disturbance.


Copy/paste error in docs for uap:FileTypeAssociation

The first property here has the following description:
"DesiredView: The desired amount of screen space to use when the appointment launches."
Huh? This tag has nothing to do with appointments, right? Looks like a copy/paste error from some other documentation.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b10bedc7-7a98-c539-8bb6-cc7d258ec400
Version Independent ID: f7cde4ec-6c81-bf49-458c-42e8521a1191
Content: uap:FileTypeAssociation (Windows 10) - Windows UWP applications
Content Source: winrt-related-src/schemas/appxpackage/uapmanifestschema/element-uap-filetypeassociation.md
Product: uwp
Technology: winrt-reference
GitHub Login: @drewbatgit
Microsoft Alias: drewbat

Thanks for catching this. The description has been updated to the following:
"DesiredView = 'An optional string that can have one of the following values: "default", "useLess", "useHalf", "useMore", or "useMinimum".'"


[VEN-1324]: refactor maxLoops check for enterMarkets and add rewardDistributor met…

…hods.
Description

Resolves #
Checklist

[ ] I have updated the documentation to account for the changes in the code.
[ ] If I added new functionality, I added tests covering it.
[ ] If I fixed a bug, I added a test preventing this bug from silently reappearing again.
[ ] My contribution follows Venus contribution guidelines.

https://github.com/VenusProtocol/isolated-pools/pull/202


Mechanism to pass additional attributes to custom eval or loss functions

Hello Team,
Great work on the package. I know there is a support for custom eval and loss function which is awesome. However, is there a way to pass additional attribute(s) (maybe part of feature set or maybe not). For example, imagine if there is a certain attribute by which we want the parity in the model's performance, then mathematically it's easy to create a custom eval function, but the function will need more data than just the predictions and labels. So, the question is, does there such mechanism exist in LightGBM?
Best,
Jayant

Thanks for using LightGBM and opening this issue.
Could you be more specific about what types of parameters you'd like to pass? We might be able to suggest a way to accomplish what you'd like to accomplish without needing to change the way lightgbm handles custom metric / objective functions.

Yes, definitely. For example, let's say my training data has a column/feature which take values 'A', 'B', and 'C' and I want to make sure that the False Positive Rate on the evaluation data (which I use for early stopping criteria) is same across all values that feature can take.

@jayantsahewal I think you can use train() function from standard API. fobj and feval arguments of that function are callables which take whole Dataset object via train_data argument. So you can compute everything you want from original features. Just don't forget to specify free_raw_data=False during Dataset creation.

@StrikerRUS I think we cannot get the origin feature values from train_data in python directly? A dirty work around would be like this, which explicitly record the original feature value in python and directly use it in the evaluation function.
X = np.array([[1, 2, 3], [4, 5, 6]])
y = np.array([0, 1])
train_data = lgb.Dataset(data=X, label=y)

def custom_eval_func(preds, train_data):
    return "nonsense", np.mean(X[:, 0] + preds), True

params = {'objective':'regression'}
lgb.train(params=params, train_set=train_data, valid_sets=[train_data], valid_names=["train"], feval=custom_eval_func)

@shiyu1994 I believe we can. Just as  I mentioned above, free_raw_data should be False. Please consider checking the following snippet (reworked your code):
import lightgbm as lgb
import numpy as np

X = np.array([[1, 2, 3], [4, 5, 6]])
y = np.array([0, 1])
train_data = lgb.Dataset(data=X, label=y, free_raw_data=False)

def custom_eval_func(preds, train_data):
    return "nonsense", np.mean(train_data.get_data()[:, 0] + preds), True

params = {'objective': 'regression'}
lgb.train(params=params, train_set=train_data, valid_sets=[train_data], valid_names=["train"], feval=custom_eval_func)

Sorry. I did not noticed the get_data() method.

@shiyu1994 No problem! Dataset class is really "fat", it is impossible to remember all methods.
@jayantsahewal Does get_data() method suit your needs?


Zero value of feature ID is changed to Undefined

One of my rendered feature has intentionally ID set to 0. Since 3.10.1 when this feature is retrieved using vectorLayer.getSource().getFeatures(), that zero value ID is converted into 'Undefined'. Should I avoid such values or is it a newly introduced bug in OL?

This is a bug, introduced with the removal of goog.isDef and goog.isNull. Thanks for reporting.

Thanks a lot for quick fix!


does not work

0 info it worked if it ends with ok
1 verbose cli [ 'C:\Program Files\nodejs\node.exe',
1 verbose cli   'C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js',
1 verbose cli   'run',
1 verbose cli   'test:single' ]
2 info using<EMAIL_ADDRESS>3 info using<EMAIL_ADDRESS>4 verbose run-script [ 'pretest:single', 'test:single', 'posttest:single' ]
5 info lifecycle grow@1.1.0~pretest:single<EMAIL_ADDRESS>6 silly lifecycle grow@1.1.0~pretest:single: no script for pretest:single, continuing
7 info lifecycle grow@1.1.0~test:single<EMAIL_ADDRESS>8 verbose lifecycle grow@1.1.0~test:single: unsafe-perm in lifecycle true
9 verbose lifecycle grow@1.1.0~test:single: PATH: C:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin;D:\work\grow\node_modules.bin;C:\Program Files\nodejs;C:\Users\Oleksandr\bin;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\local\bin;C:\Program Files\Git\usr\bin;C:\Program Files\Git\usr\bin;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Users\Oleksandr\bin;C:\Program Files (x86)\Intel\iCLS Client;C:\Program Files\Intel\iCLS Client;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\Intel\WiFi\bin;C:\Program Files\Common Files\Intel\WirelessCommon;C:\Program Files (x86)\GtkSharp\2.12\bin;C:\Program Files\nodejs;C:\Program Files\Git\cmd;C:\Program Files\Android\Android Studio;C:\Users\Oleksandr\AppData\Local\Android\sdk;C:\Users\Oleksandr\AppData\Roaming\npm;C:\Program Files\Java\jdk1.7.0_79\bin;C:\Users\Oleksandr\AppData\Local\atom\bin;C:\Program Files\Git\usr\bin\vendor_perl;C:\Program Files\Git\usr\bin\core_perl
10 verbose lifecycle grow@1.1.0~test:single: CWD: D:\work\grow
11 silly lifecycle grow@1.1.0~test:single: Args: [ '/d /s /c', 'istanbul cover src/index.test.js' ]
12 silly lifecycle grow@1.1.0~test:single: Returned: code: 1  signal: null
13 info lifecycle grow@1.1.0~test:single: Failed to exec test:single script
14 verbose stack Error<EMAIL_ADDRESS>test:single: istanbul cover src/index.test.js
14 verbose stack Exit status 1
14 verbose stack     at EventEmitter. (C:\Program Files\nodejs\node_modules\npm\lib\utils\lifecycle.js:245:16)
14 verbose stack     at emitTwo (events.js:106:13)
14 verbose stack     at EventEmitter.emit (events.js:191:7)
14 verbose stack     at ChildProcess. (C:\Program Files\nodejs\node_modules\npm\lib\utils\spawn.js:24:14)
14 verbose stack     at emitTwo (events.js:106:13)
14 verbose stack     at ChildProcess.emit (events.js:191:7)
14 verbose stack     at maybeClose (internal/child_process.js:852:16)
14 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:215:5)
15 verbose pkgid<EMAIL_ADDRESS>16 verbose cwd D:\work\grow
17 error Windows_NT 10.0.10586
18 error argv "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js" "run" "test:single"
19 error node v6.2.2
20 error npm  v3.9.5
21 error code ELIFECYCLE
22 error<EMAIL_ADDRESS>test:single: istanbul cover src/index.test.js
22 error Exit status 1
23 error Failed at the<EMAIL_ADDRESS>test:single script 'istanbul cover src/index.test.js'.
23 error Make sure you have the latest version of node.js and npm installed.
23 error If you do, this is most likely a problem with the grow package,
23 error not with npm itself.
23 error Tell the author that this fails on your system:
23 error     istanbul cover src/index.test.js
23 error You can get information on how to open an issue for this project with:
23 error     npm bugs grow
23 error Or if that isn't available, you can get their info via:
23 error     npm owner ls grow
23 error There is likely additional logging output above.
24 verbose exit [ 1, true ]

Not sure I understand the logs, but I might have the same problem.
Istanbul + karma used to work fine one node 6.1, I've now upgraded to 6.3 and my tests are broken.
Tests pass if I disable istanbul

Sorry turn out that my problem was coming from a weird combination of rollup plugins, for some reason the code to be instrumented ended up being different between node 6.1 and 6.3, black magic...
Ignore me

Failed at the<EMAIL_ADDRESS>test:single script 'istanbul cover src/index.test.js'.
Are you sure this isn't an issue with the grow project?


OpenVPN issues config file generation

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:

[x] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
[x] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug
A.)
If a certificate revocation list ist selected in settings under instances [new] the openvpn will not start.
The generated config file will point to a file named /var/etc/openvpn/server-[instance-id].crl-verify but the file is not there.
The logfile shows:
2023-08-03T21:01:24   Notice   openvpn_server10   Options error: --crl-verify fails with '/var/etc/openvpn/server-[instance-id].crl-verify': No such file or directory (errno=2)
B.)
The drop-down "Certificate Depth" ist completely ignored. If you save the instance and reopen it, it will always show "Do Not Check".
To Reproduce
Steps to reproduce the behavior:

Go to VPN, OpenVPN, Instances [new]
Click on 'OpenVPN'
Add a Instance
Select "Certification Revocation List" -> "RevocationsList"
Save and Apply

Environment
2x OPNsense 23.7 (amd64, OpenSSL) as HA
Client Windows 10 / Firefox 115.0.2 x64

The depth thing is highly likely https://github.com/opnsense/core/commit/68ae7bd48f1a64491fc3cb97da95283bad0ba6d7

I tested it. this didn't work but
command:/usr/local/sbin/pluginctl -c crl && /usr/local/opnsense/scripts/openvpn/ovpn_service_control.php
will do.

@hsiewert you need to restart configd before this will work, (service configd restart)

@AdSchellevis confirmed, retested, you are right. Sorry.

@hsiewert thanks for confirming!


Updates certificate source

Changes certificate source from a directory to a file representation. Currently it won't upload anything because the source is not a folder.

@Toilal Any chance of getting this merged and released? It fixes the problem we're seeing as well.

Hi,
Toilal does not work for the company anymore, and I'm trying to figure out what those projects do.
Could you please give me a little bit of context to better understand the "why" of the change ?
Are you still interested into merging this after all those years ?


pull request for e2e test at 01/19/2017 04:56:11

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Go to the next waterfallstep without prompt.

How can i move to the next step? Bot emulator is saying "couldn't send retry".
I want the bot to continue to the next step if the user chooses "Near me" and
to prompt the user if the user chooses "Somewhere else" before moving to the next step.
Thank you.
            AddStep(async (stepContext, cancellationToken) =>
            {
                var realEstateType = stepContext.Result as FoundChoice;
                var state = await (stepContext.Context.TurnState["FPBotAccessors"] as FPBotAccessors).FPBotStateAccessor.GetAsync(stepContext.Context);
                state.RealEstateType = realEstateType.Value;

                return await stepContext.PromptAsync("choicePrompt",
                   new PromptOptions
                   {
                       Prompt = stepContext.Context.Activity.CreateReply("Which location are you considering?"),
                       Choices = new[] {new Choice {Value = "Near me"},
                                        new Choice {Value = "Somewhere else"}
                       }.ToList()
                   });

            });

            AddStep(async (stepContext, cancellationToken) =>
            {
                var nearOrSomewhereElse = stepContext.Result as FoundChoice;
                var state = await (stepContext.Context.TurnState["FPBotAccessors"] as FPBotAccessors).FPBotStateAccessor.GetAsync(stepContext.Context);
                state.NearOrSomewhereElse = nearOrSomewhereElse.Value;

                if (state.NearOrSomewhereElse != "Near me")
                {
                    return await stepContext.PromptAsync("textPrompt",
                       new PromptOptions
                       {
                           Prompt = stepContext.Context.Activity.CreateReply("Which location are you considering?")
                       });
                }
                return await stepContext.ContinueDialogAsync(cancellationToken);
              //return await stepContext.NextAsync();
            });

            AddStep(async (stepContext, cancellationToken) =>
            {
                var nearOrSomewhereElse = stepContext.Result as FoundChoice;
                var state = await (stepContext.Context.TurnState["FPBotAccessors"] as FPBotAccessors).FPBotStateAccessor.GetAsync(stepContext.Context);
                state.NearOrSomewhereElse = nearOrSomewhereElse.Value;

                return await stepContext.PromptAsync("choicePrompt",
                 new PromptOptions
                     {
                       Prompt = stepContext.Context.Activity.CreateReply($"Please indicate the size of {state.RealEstateType}"),
                       Choices = new[] {new Choice {Value = "Size 1"},
                                        new Choice {Value = "Size 2"},
                                        new Choice {Value = "Size 3"}
                       }.ToList()
                 });

            });`

The Microsoft Bot Framework team prefers that how to questions be submitted on Stack Overflow. The official Bot Framework Github repos are the preferred platform for submitting bug fixes and feature requests.


[DP-172] Add home page for CWKW collection at "/cwkw"

Changelog
Moved all CWKW related files into website/src/pages/cwkw folder. Also added a TOC sidebar with dummy data for now. See images below of the new front-end changes.

index.page.tsx: Starting home page of CWKW collection.

Added a prototype collapsible table of contents sidebar to the left side of the screen (I don’t think this will be on the home page in the actual deployment, but I added this to show what I thought the TOC could look like.)

index.page.css.ts: Added some styles in consideration of a collapsible TOC panel, such as shifting the main content to the right side of the screen.
theme.css.ts: Just modified a few colors here so it’s more readable, but I assume this won’t be finalized yet.
toc-sidebar.tsx:

Created a desktop panel and mobile panel view.
Added styling with toc-sidebar.css.ts.

Sidebar when closed:

Sidebar when open:

Mobile sidebar when closed: (For now, I made the button to open the drawer a button at the bottom right of the screen since I wasn't really sure where to put it... But this can be changed, I'm really open to suggestions)

Mobile sidebar when open:

Since we decided not to have a menu apart from the table of contents, for mobile the menu button at the top can open it.

If this home page is gonna have the TOC as part of the page, i'm thinking it won't need the TOC as a drawer as well.

Changelog

index.page.tsx: TOC drawer starts out expanded, and can be collapsed.

cwkw-layout.tsx:

Menu items are removed.
On mobile, the drawer icon can be seen and opened to reveal the TOC drawer.

toc-sidebar.tsx:

Removed mobile icon from the bottom right.

Changelog

toc.page.tsx and cwkw-layout.tsx: Replaced usage of my own <Chapters /> component with preexisting <TOC />.
toc-sidebar.css.tsx: Made sure mobile nav button at top left shows when TOC desktop sidebar no longer does.

Current Appearance of TOC (Need to change colors later)

On mobile:

Picture of nested chapters content showing in TOC:


[Bug] Exit animations in Solid.js >= 1.5.0 don't work

1. Describe the bug
Exit animations in Solid.js >= 1.5.0 don't work. It was released recently so wasn't really expecting it to work but just thought it was worth flagging.
2. IMPORTANT: Provide a CodeSandbox reproduction of the bug
https://codesandbox.io/s/sandpack-project-forked-78xcg2
3. Steps to reproduce
Click on toggle and watch it animate in and not animate out.
You can change solid.js to version 1.4.x and it will work again.
4. Expected behavior
Should animate out

batching behavior changed in 1.5
https://playground.solidjs.com/?hash=-423275382&version=1.4.8
https://playground.solidjs.com/?hash=1236122958&version=1.5.3
Basically it should work if these lines are swapped
https://github.com/motiondivision/motionone/blob/24292336e5c7f797e9fbfa6e42160337968a9afe/packages/solid/src/presence.tsx#L92-L93
https://github.com/motiondivision/motionone/blob/24292336e5c7f797e9fbfa6e42160337968a9afe/packages/solid/src/presence.tsx#L97-L98
with new batching, el() here will get the new value updated beforehand.
https://github.com/motiondivision/motionone/blob/24292336e5c7f797e9fbfa6e42160337968a9afe/packages/solid/src/presence.tsx#L120

@mdynnl Your fix here works for me! I think you should open a PR to get this in.

This should be fixed in 10.14.2!


Decouple activation to layer

Might be related to #152.
AS-IS
Current implementation of fc_layer has activation function and loss function coupled.
TO-BE
IMO, decoupling the activation layer as well as loss function(as @kparichay noted) is also needed to increase modularity.
for example in Keras, because activation functionality is decoupled, putting bn_layer between fc and activation can be done easily.
model.add(Dense(64))
model.add(BatchNormalization())
model.add(Activation('tanh'))

still activation can be used as an argument of fc_layer possible because fc_layer can have activation_layer
model.add(Dense(64, activation="relu" ))

Proposal

Add activation_layer(or equivalent) and extract the functionality.
Change fc_layer to have activation_layer

@jijoongmoon Could you confirm if this change is okay?

:octocat: cibot: Thank you for posting issue #153. The person in charge will reply soon.

I thinks it must be done soon. Thanks for addressing this issue.


Generate the name of a master using family, width, weight, custom order

Fixes https://github.com/googlei18n/glyphsLib/issues/323.

does the _splitName function also need to be adjusted accordingly now that you changed the order returned by _joinName?
also, what does Glyphs.app when generating UFO masters? Does it write the width before the weight?

Jany says that _splitNames does not expect order.

ok thanks. So was change this unintentional, or has something else changed in Glyphs.app recently?

Jany says that _splitNames does not expect order.

perhaps for consistency with _joinNames, _splitNames should return the tuple as (width, weight, custom)
https://github.com/googlei18n/glyphsLib/pull/333/files#diff-2486ae3fe6d67c56b1c1b22402d779c1R1386

can we merge this?

So was this change unintentional, or has something else changed in Glyphs.app recently?

Unintentional, I was confused by the fact the elsewhere in the code, the weight is handled before the width, but didn't think of how the stylenames should look in the end.

perhaps for consistency with _joinNames, _splitNames should return the tuple as (width, weight, custom)

Bof, I don't know, both those methods are supposed to be private so it does not really matter

can we merge this?

Yes


gfxrecon-convert: Emit index field when writing metadata JSON blocks

This PR is to fix issue https://github.com/LunarG/gfxreconstruct/issues/1343 where gfxrecon-convert would not emit indices for metadata blocks.

Author nickdriscoll-lunarg not on autobuild list. Waiting for curator authorization before starting CI build.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Hey, Panos, could you take a look at this PR?  Thank you!

CI gfxreconstruct build # 3733 running.

CI gfxreconstruct build # 3733 failed.

CI gfxreconstruct build queued with queue ID 134575.

CI gfxreconstruct build queued with queue ID 134580.

Panos,
I have made the suggested edits and it looks like its building everywhere, though I'm not sure why the final check won't turn green. I did accept the CLA.
Let me know if this still needs work.
Thank you.

CI gfxreconstruct build # 3737 running.

CI gfxreconstruct build # 3737 passed.

CI gfxreconstruct build queued with queue ID 135750.

CI gfxreconstruct build # 3739 running.


Error: 'ECONNREFUSED'

The logs:
2022-10-13T01:43:01.712Z INFO devtools:puppeteer: Initiate new session using the DevTools protocol
2022-10-13T01:43:01.713Z INFO devtools: Launch Google Chrome with flags: --disable-popup-blocking --disable-extensions --disable-background-networking --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-sync --metrics-recording-only --disable-default-apps --mute-audio --no-first-run --no-default-browser-check --disable-hang-monitor --disable-prompt-on-repost --disable-client-side-phishing-detection --password-store=basic --use-mock-keychain --disable-component-extensions-with-background-pages --disable-breakpad --disable-dev-shm-usage --disable-ipc-flooding-protection --disable-renderer-backgrounding --force-fieldtrials=*BackgroundTracing/default/ --enable-features=NetworkService,NetworkServiceInProcess --disable-features=site-per-process,TranslateUI,BlinkGenPropertyTrees --window-position=0,0 --window-size=1200,900
2022-10-13T01:43:02.245Z INFO devtools: Connect Puppeteer with browser on port 32054
FetchError: Failed to fetch browser webSocket URL from http://localhost:32054/json/version: request to http://localhost:32054/json/version failed, reason: connect ECONNREFUSED ::1:32054
    at ClientRequest.<anonymous> (C:\Users\joshu\AppData\Roaming\npm\node_modules\archiver-for-google-keep\node_modules\node-fetch\lib\index.js:1491:11)
    at ClientRequest.emit (node:events:513:28)
    at ClientRequest.emit (node:domain:489:12)
    at Socket.socketErrorListener (node:_http_client:494:9)
    at Socket.emit (node:events:513:28)
    at Socket.emit (node:domain:489:12)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  type: 'system',
  errno: 'ECONNREFUSED',
  code: 'ECONNREFUSED'
}

Screenshot:

Hi @JoshuaGarrison27 I do recall that on occasion I would start this up and it would fail to connect to the browser.  Simply CTRL-Cing out of it and restarting generally worked.  If It's persistent, then I'll need to dig in and see if I can replicate.
In the meantime, you might try using Chromedriver.  WebdriverIO can use two different protocols to connect to Chrome, devtools is one, and WebDriver (via Chromedriver) is the other.  Do you want to try with WebDriver protocol instead, in the meantime?
I also see you're on Windows. I haven't tried on this OS yet.  Maybe there's some Windows firewall thing getting in the way?
Here's the instructions for using Chromedriver instead:  See Other Protocols and Options:
Install Chromedriver's latest version:
$ npm install chromedriver@latest

Then start chromedriver listening on port 4444:
$ node_modules/chromedriver/bin/chromedriver --port=4444

In another terminal, you should then be able to start the archiver tool:
$ KEEP_DATA_DIR=./webdriver-data-dir npm start

Regarding the bug, my options are either find a way to bundle in chromedriver to avoid needing to manually start it, or report the devtools protocol issue to WebdriverIO and see if they can fix the problem.
If this doesn't work, another idea is you can try running it in a Linux Docker container, such as https://hub.docker.com/r/selenium/standalone-chrome.  You can interact with the Linux desktop via port 5900 via a VNC client, or http://localhost:7900 via noVNC in your browser.
Thanks for trying it out!

I forgot to come back to this. I got it to finally run. I made a mistake when I got the windows firewall popup or something. After tinkering with several settings I finally got it to go through. I didn't make good notes so I cant give any input in that regard. I'll leave it up to you if you want to close this or not. In the end, issue was windows firewall.

I thought it might have something to do with the firewall.  Glad you got it working.  I'll leave this open because I think it would be helpful to detect firewall problems and then do extra logging like Did you check your firewall? . It would help point others in the right direction.


Layout/RescueEnsureAlignment not always correctly detecting alignment (again)

Expected behavior
Layout/BeginEndAlignment should correctly detect begin/rescue/end alignment.
Actual behavior
If begin/rescue/end is assigned to value then the alignment is being incorreclty detected:
Steps to reproduce the problem
Given this code:
results = begin
            client.search(q: query,
                          index: index,
                          from: cursor,
                          size: per_page,
                          sort: sort).dig('hits', 'hits').to_a
          rescue StandardError
            []
          end

elasticsearch.rb:224:21: C: Layout/RescueEnsureAlignment: rescue at 224, 20 is not aligned with results = begin at 218, 10.
                    rescue StandardError
                    ^^^^^^

Autocorrected to this:
results = begin
            client.search(q: query,
                          index: index,
                          from: cursor,
                          size: per_page,
                          sort: sort).dig('hits', 'hits').to_a
rescue StandardError
  []
          end

RuboCop version
$ [bundle exec] rubocop -V
0.91.0 (using Parser <IP_ADDRESS>, rubocop-ast 0.4.0, running on ruby 2.7.1 x86_64-linux)

Related Issues

Probably same as #8710

Can confirm fixed in master by #8719


Possible typo ? (PGDARA instead of PGDATA)

There is a possible typo at https://github.com/neondatabase/neonvm/blob/af11b744ccae0d8640ea221fb5987d73f30b0175/samples/vm-example/Dockerfile#L30
The PGDARA variable cannot be found anywhere else in your repositories or in postgresql's documentation.
This PR modifies the line as follows:
https://github.com/neondatabase/neonvm/blob/bae65d67bcab2819d6c3f156d8db4183133341e1/samples/vm-example/Dockerfile#L30

Good catch, thank you!