Picture Clause representation

I am getting confused by how a picture clause is defined in the data structure and what should and should not be considered an appropriate representation.
Taking into account the following said in the paper:

More specifically that last sentence.
The way I understand it, the representation can be almost anything. And if any characters which are not part of the semantics are encountered, then they are taken as their exact value.
E.g. We could have "PICTURE IS XXCXX" which would represent any 2 characters followed by a char 'C' and then again any 2 characters.
Another question that came up is whether Fields composed of both digits and characters can have leading digits or signs.
Are the 2 following representations valid? 1)"S99999V999XXXXXX" and 2) "SXVXX999" where in the first example, the S and V become part of the digit representation of the number and in the second, they are part of the String representation and if soo, is their representation the character 'S', 'V' or is it '+/-', ' , / . '.

I also have a follow up question about this: if characters and digits are allowed together in a picture clause, say "9X9X", does this mean that a field represented by such a picture clause would store a string of 4 characters where the 1st one is a digit char (e.g. '1' or '3'), the 2nd one is any character (including digit chars), the 3rd one a digit char again, etc.? Is it also true that upon storing any value in such a field, that value must obey the picture clause of the field?

In practice, COBOL developers usually do not mix 9s (and Zs, Vs, Ss, etc) with Xs (or As) and thus keep "types" either numeric or non-numeric. For your implementation it would be most logical to choose to pursue one of the two clean paths:

either go strictly character by character and stick to that — which is straightforward but might lead to awkward internal representation of runtime values, and will still require some "global" constraint checks (e.g., if there is only one V among the 9s or that Zs are not in the middle of 9s)
or parse each picture pattern into either a purely numeric one (and then represent it with either numeric types natural to your target language/VM, or at least use one uniform implementation), or a non-numeric one; in that case for examples you posted above, you can decide to either issue an error or just straight fall back to string: if dev's PICTURE pattern does not make sense as a proper fully numeric one, then that's it, all 9s are literally fixed characters "9", etc.

The paper about BabyCobol mentions the following: "In modern high-level programming languages the closest analogy to a picture clause would be an object field with a getter ... and a setter that parses the provided value according to the same pattern.".  Does this mean that 2 strings with different picture clauses but same amount of characters, say X PICTURE IS 123X(5) and Y PICTURE IS 456X(5), are different because their picture clauses are different, or they are considered equal as long as the 5 characters stored in them are the same? In other words, do any extra characters that are not part of the picture clause characters influence variable assignments and other operations (such as string concatenation: X + Y)?

PICTURE clauses are all about data representation. If you define X PICTURE IS 123X(5), you are stating that this field will take up eight bytes, the first three of which are fixed and the other five can be anything depending on the values.
By definition, no valid value of X will ever be equal to any valid value of Y.
EVALUATE X + Y
    WHEN "123HELLO456WORLD"
        DISPLAY "QAPLA'"
END.

(To prepare for this, you also need to include 123 in the preceding MOVE statements. Think of the extra characters as units: if you define your MONEY IS PICTURE $999V99, then you don't want someone just executing the assignment MOVE €20 TO MONEY without raising an error or at least a warning)

PICTURE clauses are all about data representation. If you define X PICTURE IS 123X(5), you are stating that this field will take up eight bytes, the first three of which are fixed and the other five can be anything depending on the values.
By definition, no valid value of X will ever be equal to any valid value of Y.
EVALUATE X + Y
    WHEN "123HELLO456WORLD"
        DISPLAY "QAPLA'"
END.

(To prepare for this, you also need to include 123 in the preceding MOVE statements. Think of the extra characters as units: if you define your MONEY IS PICTURE $999V99, then you don't want someone just executing the assignment MOVE €20 TO MONEY without raising an error or at least a warning)

Just a sanity check: the example for move makes sense, but then, all numeric variables need to have the same picture to be used in a clause? e.g. ADD X TO Y GIVING Z, do X, Y and Z need to have  the exact same picture, say: $999? If yes, does that also apply to comparisons, e.g. using ==, or in that case it's enough to check the numeric value, and forget about the $ sign?


add IE11 browser detection regex

IE11 has 'Trient' instead of 'MSIE' in navigator.userAgent string.
so I want to merge this PR
https://github.com/srfrnk/ng-device-detector/pull/20
BUT I can't fixed BROWSER_VERSIONS_RE_MAP when Brower IE Trient version.
Could you check is also?

Hi! @srfrnk
I add IE11 in my local machine Test case and request PR
https://github.com/srfrnk/ng-device-detector/pull/22
BUT a Point of concern is that modified BROWSER_VERSION_RE_MAP in ng-device-detector.js
each Map value was regex, except IE has array
I'm not sure whether is correct~
Thanks you

Can you create a tag for this? I'm trying to pull in 1.1.2 via bower but getting an error that the version isn't available (since there isn't a tag for it).

@derekmooney  - Give it a try now please... LMK if you've still got a problem.

Yeah, all good, thanks!


Performing generic requests (for other services)

Is it currently possible to perform Amazon requests for other services or would this be a worthwhile update to the library?
My use-case for this is signing requests to Amazon Elasticsearch which (as far as I can tell) is not supported by ex_aws. I would prefer to limit the access to the ES domain to a specific IAM Role for security and to do this my requests need to be signed. Signing is a non-trivial process which is obviously allready implemented in the ex_aws library so I would like to make use of this capability.
I could not find an obvious (documented) way to do this currently. Did I overlook this? Is this something we could/should add?

Hey!
This is a good question, and you're right that there is not documentation on the subject. The good news is that it is generally quite possible.
Services are categorized by operation type. Service modules (like Dyanamo or S3) are really just nice functions for building structs of a given operation type. There's really only about a half dozen total operation types.
Provided that we already have the operation type used by the service you're looking for, you're free to just build a struct with whatever parameters sit on it you like, and do operation |> ExAws.request.
Elasticsearch is a RestJSON type operation, which we do not have at this moment. However we do have a JSON type operation and I think I could add RestJSON here pretty quickly. I need to do so for one of the PRs that was recently submitted as well.
Let me leave this open for the moment and see if I can get that added here in short order.

Good to hear, if I can help you in any way with implementing this just let me know

Hey! Per the updates on master it should be quite possible to simply build a particular operation struct manually. Do note that you will need to supply any config necessary for the service to the ExAws.request functions.


How can we add support for Token.Jwt(5.1.2) or update to IdentityModel.Protocols.WsFederation 2

It is currently with 4.0.4, which makes difficult to integrate with IdentityModel.Protocols 2.1.2

I'd welcome a pull request. I have a feeling that there would be some big dependency conflicts.

Yes there quite as in version 2, system.identitymodel.protocol.extensions decoupled to microsoft.identitymodel.tokens and protocol
Also, version 1 used JWT 4.0.4 where as version 2 used >= 5.1.2
I will check for PR, but feels like need to wait for WSFederation beta release.


[QUESTION] Updating Sub document

Is there a way to update or add a sub object without getting the whole array?
Or do you need to put everything in separate collections? And make simple objects? I may have thousands of objects in the array, and if I add or remove one of them, I don't want to get all

@Avatarchik If you have large arrays and intend to frequently include and remove objects from it, it is recommended to store these objects in their own collection, keeping only a reference in the original object.


LSTM example for GPU delegate

Hello,
I am trying to run my model in android GPU using TF lite and gpu delegate.
But I faced this "UNIDIRECTIONAL SEQUENCE LSTM: Operation is not supported".
From the document "https://www.tensorflow.org/lite/performance/gpu_advanced", LSTM v2 is supported now.
All examples in tensorflow git are not using this LSTM.
Is there any example for "LSTM v2 (Basic LSTM only)" ?

My model is using following
tf_enc_model = tf.keras.models.Sequential([
tf.keras.layers.Input(shape=(1, input_dim), name='input'),
tf.keras.layers.LSTM(hidden_size, time_major=False, return_sequences=True)

@ymodak  Hello, Can you show me any document or example for LSTM gpu delegate ?

TFLite GPU doesn't have a good story for RNNs.
First of all, only the basic LSTM is supported.  Unidirectional, bidirectional, full LSTMs are not implemented and we don't have immediate priorities to add the missing ops.
Then comes the problem around invoking the LSTM network with a for loop.  What we've seen in the past is the following pattern:
for (int i = 0; i < n; ++i) {
  // copy output state from i - 1 to input state to i
  interpreter->Invoke();
}

Unfortunately, this will take up a LOT of time in GPU/CPU synchronization, and at that point, you might as well just stay in the CPU land.
Now a terrible workaround that I have done in one of the LSTM projects is the manual unrolling.  Instead doing the for loop programmatically 28 times at Invoke, I've replicated the graph structure 28 times in the TFLite model, so that it is a single Invoke call.


Fix/353 (invalidating dep cache)

The implementation of invalidating dependency cache mentioned in #353. Works like this:

the cache manager keeps a set of ids of nodes with dirty cache (DIRTY);
during each dependsOn call, the manager firstly checks whether the node's cache is dirty; in case it is, the manager re-collects dependencies of the node and removes it from the set of dirty nodes;
for each node P the cache manager keeps a set of ids of (direct and indirect) children CHILDREN_WITH_CACHE(P.id) that have P.id in the dep cache;
when deps of node P are changed, it and all its children from the CHILDREN_WITH_CACHE(P.id) set are added to the DIRTY set; after that the set CHILDREN_WITH_CACHE(P.id) is emptied;
the cache of a new node is considered dirty.

Collecting dependencies:

start with an empty set FLAT_DEPS(N) where N is the node of interest;
add ids of all direct dependencies to the set FLAT_DEPS(N);
for each direct dependency DIRECT_DEP, get (or recursively collect, if needed) ids of it's deps FLAT_DEPS(DIRECT_DEP) and add to the set FLAT_DEPS(N);
for each dependency DEP in the set FLAT_DEPS(N), add N.id to the set CHILDREN_WITH_CACHE(DEP.id).

The pros:

it is fast when dependency graph is immutable (costs one extra hashtable lookup during each dependsOn call);
only the affected subgraph is re-calculated after changes of dependencies;
dependencies of each node in the affected subgraph are calculated only once between graph mutations;
dependencies are calculated lazily;
memory is not leaking because cache invalidation mechanism deals only with ids of nodes.

The cons:

it has a little extra cost compared to the immutable cache (see perf. test results);
each Observable exports the flatDeps property that is visible from the outside but intended for internal use only; maybe it should be prefixed with underscore.

Performance tests:

master (immutable cache)

diamond
24.12 ops/sec
±6.97% (36 runs sampled)

combo
38.64 ops/sec
±10.51% (39 runs sampled)

zip
1,412 ops/sec
±9.16% (84 runs sampled)

Bacon.combineTemplate.sample
321 ops/sec
±5.44% (82 runs sampled)

Bacon.combineTemplate (deep)
18.25 ops/sec
±4.51% (37 runs sampled)

Bacon.combineTemplate
212 ops/sec
±3.48%  (86 runs sampled)

EventStream.map
3,402 ops/sec
±5.07% (83 runs sampled)

EventStream.scan
3,142 ops/sec
±4.93% (83 runs sampled)

EventStream.toProperty
3,217 ops/sec
±4.60% (81 runs sampled)

fix/353 (without cache)

diamond
9.85 ops/sec
±6.66% (30 runs sampled)

combo
39.89 ops/sec
±7.57% (40 runs sampled)

zip
1,445 ops/sec
±8.67% (85 runs sampled)

Bacon.combineTemplate.sample
325 ops/sec
±5.18% (82 runs sampled)

Bacon.combineTemplate (deep)
18.03 ops/sec
±5.56% (40 runs sampled)

Bacon.combineTemplate
208 ops/sec
±3.52% (86 runs sampled)

EventStream.map
3,411 ops/sec
±4.46% (84 runs sampled)

EventStream.scan
3,087 ops/sec
±4.13% (88 runs sampled)

EventStream.toProperty
3,154 ops/sec
±3.75% (88 runs sampled)

fix/353 PR (invalidating cache)

diamond
19.44 ops/sec
±6.46% (38 runs sampled)

combo
41.62 ops/sec
±4.23% (52 runs sampled)

zip
1,330 ops/sec
±3.95% (86 runs sampled)

Bacon.combineTemplate.sample
313 ops/sec
±4.19% (83 runs sampled)

Bacon.combineTemplate (deep)
17.75 ops/sec
±3.25% (49 runs sampled)

Bacon.combineTemplate
208 ops/sec
±2.75% (85 runs sampled)

EventStream.map
3,216 ops/sec
±4.15% (85 runs sampled)

EventStream.scan
3,125 ops/sec
±4.30% (86 runs sampled)

EventStream.toProperty
3,144 ops/sec
±4.13% (89 runs sampled)

There are many possible optimizations to this impl, but I think that the current performance is quite acceptable.

Maybe we should introduce a single internal state field like bacon_state into Observables and collect all the ugly details there.

Isn't this a book example for when to use WeakMaps? You could have a WeakMap as the shared object with the Observables as keys, and thr WeakMap will lose its entry for an Observable as soon as no other reference "remembers" that Observable. (At that point the entry can safely be dropped because it couldn't be accessed anymore anyway.)

@mk-pmb this issue is no longer relevant, as I redesigned the "glitch-prevention system" to work without a cache a couple of years ago in commit 65b6e673d72993d454649a6d305eac3c4ab423c2
You can find the current implementation at https://github.com/baconjs/bacon.js/blob/master/src/updatebarrier.js


Equatable conformance

Question
Hi, it's possible to add Equatable conformance to generated models? Adding it post factum prevents automatic synthesis
Extension outside of file declaring struct 'categoryListPayloadPayload' prevents automatic synthesis of '==' for protocol 'Equatable'

Hi @cpd,
all the generated types should already be Equatable (because they're Hashable, which implies Equatable as well). Are you seeing some types not being Equatable?


chore: improve table-of-contents

if headings is empty, then no generates table-of-contents

This is a pretty breaking change that we cannot merge. We have another issue that discusses a proper solution to this. Currently the workaround if people really like to check for this is to do a {{ if eq (strings.Count "li" .TableOfContents) 0 }} or something.


warn / bail when invalid paths are used in rule.include

Feature request
It happens sometimes that invalid paths are entered (via path.resolve) in rule.include. But webpack silently "accepts" this and the loader does nothing. But the user is confused why nothing has changed. Debugging of this simple but silent error is not that easy.
What is the expected behavior?
Invalid paths in rule.include (and probably also exclude) like for example path.resolve(__dirname, 'src', 'node_modules'), should throw a warning / error so the user knows that there is something not correct.
What is motivation or use case for adding/changing the behavior?
Better DX and feedback from webpack, easier debugging.
How should this be implemented in your opinion?
Warning and optional setting to bail on invalid paths.
Are you willing to work on this yourself?
no (not much time anymore)

@DanielRuf We used regexps in include/exclude, so we can't validate your path

@DanielRuf We are using regexps in include/exclude, so we can't validate your path

include supports:

regex
string
string array (array of conditions)
function
...

the issue covers the string / string array method, not regex. Maybe that was a bit unclear from my side (path.resolve() generates a string).
https://webpack.js.org/configuration/module/#ruleinclude
https://webpack.js.org/configuration/module/#condition

@DanielRuf Why you can't validate paths on your side?

In general webpack config files can have hundreds of files, many loaders and so on. In our case only a person with Node experience would have spotted this but other developers who try to make these changes get no feedback what is wrong.

also it reduce build time.

This is why I would make this optional / opt-in via setting - and some "strict" webpack mode.
Please keep beginners, juniors, unexperienced developers in mind - we could improve the DX of webpack with such simple steps.

@DanielRuf Why you can't validate paths on your side?

In general webpack config files can have hundreds of lines, many loaders and so on. In our case only a person with Node experience would have spotted this but other developers who try to make these changes get no feedback what is wrong.

also it reduce build time.

This is why I would make this optional / opt-in via setting - and some "strict" webpack mode.
Please keep beginners, juniors, unexperienced developers in mind - we could improve the DX of webpack with such simple steps.

maybe this could make sense, but this will be a breaking change.

maybe this could make sense, but this will be a breaking change.

Not if it is like I proposed: an opt-in setting (opt-in means you have to actively enable / activate it).

I think that makes sense. We can show warnings for invalid paths passed to options with absolutePath: true, that are not output paths.

this is an issue with webpack configs resolve.aliases too.  i was going to open a documentation issue but it almost seems like a webpack util for specifying paths makes more sense e.g. webpack.resolve
path.resolve is essentially path.join.  there is module.resolve which will throw MODULE_NOT_FOUND if the target doesn't exist, but it's not exactly a drop-in replacement for path.resolve.  maybe someone with more knowledge of module.resolve and webpack would know if this is a doc issue or not?
i'm pretty dumb and need to be protected from myself sometimes. asserting paths i enter actually exist would be one of those times.


ci(*): add actions/labeler to label automatically to recognize scope of Pull Request easily

This actions/labeler will label what package scope is on each Pull Request automaticallly. After this Pull Request merged, labeler can know what package scope have been changed by directory
This action is supported by GitHub Actions officially too like actions/setup-node
Example screenshot

Expectation
I am expecting each maintainers can recognize each Pull Request's package scope by labels of it more easily.

Codecov Report

Merging #6917 (3338460) into main (75b9c57) will not change coverage.
The diff coverage is n/a.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files

@@           Coverage Diff           @@
##             main    #6917   +/-   ##
=======================================
  Coverage   41.98%   41.98%
=======================================
  Files         179      179
  Lines        7039     7039
  Branches     1421     1421
=======================================
  Hits         2955     2955
  Misses       3722     3722
  Partials      362      362

Components
Coverage Δ

@tanstack/angular-query-devtools-experimental
∅ <ø> (∅)

@tanstack/angular-query-experimental
83.62% <ø> (ø)

@tanstack/eslint-plugin-query
85.11% <ø> (ø)

@tanstack/query-async-storage-persister
48.43% <ø> (ø)

@tanstack/query-broadcast-client-experimental
∅ <ø> (∅)

@tanstack/query-codemods
0.00% <ø> (ø)

@tanstack/query-core
93.63% <ø> (ø)

@tanstack/query-devtools
4.46% <ø> (ø)

@tanstack/query-persist-client-core
57.73% <ø> (ø)

@tanstack/query-sync-storage-persister
82.50% <ø> (ø)

@tanstack/react-query
93.07% <ø> (ø)

@tanstack/react-query-devtools
0.00% <ø> (ø)

@tanstack/react-query-next-experimental
∅ <ø> (∅)

@tanstack/react-query-persist-client
100.00% <ø> (ø)

@tanstack/solid-query
84.84% <ø> (ø)

@tanstack/solid-query-devtools
∅ <ø> (∅)

@tanstack/solid-query-persist-client
100.00% <ø> (ø)

@tanstack/svelte-query
62.68% <ø> (ø)

@tanstack/svelte-query-devtools
∅ <ø> (∅)

@tanstack/svelte-query-persist-client
100.00% <ø> (ø)

@tanstack/vue-query
70.81% <ø> (ø)

@tanstack/vue-query-devtools
∅ <ø> (∅)

we already have labels for packages:

will they be re-used ?

we already have labels for packages:

will they be re-used ?

Oh thanks, I'll update it


Smart selection acts weird when first function parameter has inline JSDoc

Bug Report

🕗 Version & Regression Information
This is similar to #39618.
VS Code:
Version: 1.68.0
Commit: 4af164ea3a06f701fe3e89a2bcbb421d2026b68f
Date: 2022-06-08T11:49:57.055Z
Electron: 17.4.7
Chromium: 98.0.4758.141
Node.js: 16.13.0
V8: <IP_ADDRESS>-electron.0
OS: Linux x64 5.13.0-41-generic

💻 Code

type B = {};
type A = {
    a(/** Comment */ p0: number, /** Comment */ p1: number): string;
};

🙁 Actual behavior
Having the caret at p0 and repeated Expand Selection results in the wrong behavior we've already seen in #39618:

🙂 Expected behavior
But with the caret at p1 the behavior seems fine:

This also happens with simple type definitions like this:
let a;
let b: {
    /** Comment */ p0: number
    /** Comment */ p1: number
    /** Comment */ p2: number
};
let c;

As in this capture:


Map.! caused by worker

Tried this code:
module Main exposing (main)

import Platform

main : Program () () ()
main =
    Platform.worker
        { init = \() -> ( (), Cmd.none )
        , update = \() () -> ( (), Cmd.none )
        , subscriptions = \() -> Sub.none
        }

With these invokations:
elm make --debug src/Main.elm
elm make --optimize src/Main.elm
elm make src/Main.elm

All end with the same error:
Map.!: given key is not an element in the map

Root cause is that elm.json does not have elm-lang/json in it. Going to review the others and see if it's the same thing.


@DisjointWith({X.class}) / @Contradicts({X.class})

These issues and maybe others outline aspects of how mutually-exclusive annotation pairs (@Nullable/@NonNull and @NullMarked/@NullUnmarked) should be addressed.

#242
#109
#127

So these represent two use cases for a potential meta-annotation such as @Contradicts that could go on these pairs, which could trigger the behaviors described. #200 could provide a third.
That link could be made in either direction with the same effect, or in both directions harmlessly.
(This is just a file-and-forget, for now.)

I think there's a good case for having a mutual-exclusion annotation, such as @IncompatibleWith(A.class) @interface B {} which states that if any scope is annotated with both @A and @B that's a warning/error.
I think there's also a good case for a negation/override annotation, such as @Negates(C.class) @interface D {} which states that if an outer scope is annotated with @C and an inner scope is annotated with @D then the inner scope is treated as if annotated with @D but not annotated with @C (and vice-versa?).

Dup of #101


[Feature Request] Support CRT and Key files for Mac

I am a user of the Netflix team subscription for Insomina (super happy with it!). We use certificates internally and the easiest path is to use CRT and Key files. In the support article it mentions that this format is not available for mac. I'd love for this feature to be available! It would make life easier.

Hi @emjburns! I think this may actually be supported now, if you're on macOS High Sierra or later. Let me explain.
The certificate handling in Insomnia is delegated to the libcurl library. On the Mac build, Insomnia uses the version of libcurl installed on your system. Since macOS High Sierra, Apple now includes a version of libcurl that uses LibreSSL instead of Secure Transport which, I think no longer supports p12 certificates and only CRT/Key pairs.
Can you let me know what OS you're on and any error messages you see in the Timeline tab related to certificate handling when trying CRT/Key files?

Hi @gschier, thanks for the quick reply! I'm on macOS High Sierra (10.12.6). When I try to use the CRT/Key, I see:
* Enable automatic URL encoding
* Disable SSL validation
* Enable cookie sending with jar of 17 cookies
* Adding SSL PEM certificate
* Adding SSL KEY certificate
* Hostname in DNS cache was stale, zapped
*   Trying <IP>...
* TCP_NODELAY set
* Connected to <dnsname> (<IP>) port 7004 (#13)
* WARNING: SSL: CURLOPT_SSLKEY is ignored by Secure Transport. The private key must be in the Keychain.
* WARNING: SSL: The Security framework only supports loading identities that are in PKCS#12 format.
* SSL: Can't load the certificate "</My/CRT>" and its private key: OSStatus -25299
* Closing connection 13```

This makes me think that I should upgrade my OS?

Hmm, what does Insomnia say your Curl version is? It should say at the start of the Timeline tab.

I'll leave the decision of upgrading your OS to you but I'm pretty sure you should already have the updated Curl version on High Sierra.

Oh darn thought I pasted that line too:

Ha pasting the * accidently made that line not show up.

Lol 😄
You mentioned that you were on macOS High Sierra 10.12 but High Sierra is 10.13 (10.12 is Sierra). And, now 10.14 Mohave is out already!
I suspect upgrading your OS would work but I can't take responsibility for recommending that you do it (stuff might break for you).

Was due for an upgrade anyways, updated my OS and also Insomnia and now it is working! Oops you're right about Sierra vs High Sierra. Thank you so much for your help!!

Awesome! It looks like it's time to update the docs (and perhaps the certificate dialog).


Remove the 'sh' Python library

It has problems cleaning up subprocesses created by commands, adds much implementation complexity, and does not add a lot of value over the standard library subprocess module anyway.
Since the src/program/lwaftr/test/lib/ directory only contains this library and the test_env.py file, I am moving up the latter file and removing the directory.
Part of #412.

Yay, I'm glad to see that gone on the whole. :)


make: *** No rule to make target 'build'.  Stop.

Windows 11 WSL2 Ununtu 22.04
This is a fresh install with finally everything in place. Yet I now get a completely different error
**enli@LY5:~/1/OpenDevin$ make build
make: *** No rule to make target 'build'.  Stop.**
enli@LY5:~/1/OpenDevin$ python3 --version
Python 3.11.9
enli@LY5:~/1/OpenDevin$ poetry --version
Poetry (version 1.8.2)
enli@LY5:~/1/OpenDevin$ node --version
v18.17.1
enli@LY5:~/1/OpenDevin$

@xplorer990 can you show which version you're on with git log -n1?
It's a weird error--there's definitely a build in the makefile...

make build
Building project...
Checking dependencies...
Checking Python installation...
Python 3.11.2 is already installed.
Checking npm installation...
npm 10.2.4 is already installed.
Checking Docker installation...
Docker version 26.0.0, build 2ae903e is already installed.
Checking Poetry installation...
Poetry (version 1.8.2) is already installed.
Dependencies checked successfully.
Pulling Docker image...
Using default tag: latest
latest: Pulling from opendevin/sandbox

@xplorer990 I am using ubuntu linux, for your reference. Maybe delete OpenDevin folder and retry?

check the folder and run ls

Troubleshooting Options
For your issue.
Run make with Debug on:
Additional output from make can help:
make SHELL='/bin/sh -x' build

Will display everything that make is trying to do.  This can help you resolve the issue yourself, or definitely share that with us.
Verify non-cmake installation of make (GNU Autotools):
If you happen to run this command without an error about a missing VERBOSE flag.
make VERBOSE=1 build

You have the cmake version of make installed, or it is being referenced, which is not correct.
Check environment in WSL container:
Consider what is on your environment variables, as that is imported from the Windows host when some %USERPROFILE%\.wslconfig or /etc/wsl.conf options are changed, such as host_inteop: true which can irregularly set PATH sometimes.
printenv

Advise check environment, ensure that host integration isn't pulling a windows installation of cmake from PATH before proceeding.
Your make version may also help us out:
make --version

Correct version of make is GNU auto tools make.  If you see mention of cmake here, ensure PATH isn't adding it from Windows.
Other potential hiccups (Line Endings, MSYS2 and Git Bash interference)
Weird errors can also be related to doing a git clone from within Windows, and then trying to cd to a Windows git-cloned directory in WSL, where line endings are different.  Avoid using tools like git bash, MSYS2, etc. on Windows that can interfere incorrectly with WSL2, and use git from within your WSL container as an installed package.

@loopyd, If it is an LLM response , mention it too.

@loopyd, If it is an LLM response, mention it too.

I hand type all of my responses. None of tbis is AI generated.

Cloning in new folder did the trick.


fix: explicitly add imported kitsu types to plugin

:tada: This PR is included in version 2.0.3 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Sync Log

I have created a log for whenever the synchronize button is pressed, however it is currently only documenting whether it is a complete or partial success/failure without more metadata and only storing it in the browser's local storage. Ashesh has provided me an approach on how to go about documenting this definitively in the database and I shall be working on that this week. This should be merged only after the sorting and navigation features have been merged to the main branch - this requires me to make some updates to that pull request with respect to aesthetics and status column sorting.

It'd be nice to also have source collection id and destination collection id in the SyncLog model.
model SyncLog {
  id              Int       @id @unique @default(autoincrement())
  timestamp       DateTime  @default(now())
  status          String
  detailedRecords Json
  sourceEmail     String
  sourceHost      String
  sourceCollectionID String
  destinationHost String
  destinationCollectionID String

  @@unique([timestamp, sourceHost, destinationHost])
}

We can have this in a future iteration, not necessarily required now. PR merged.


[Feature request] Support for ESP32-C6

ESP32-C6 support is available in Arduino Core ESP32 since version 3.x-alpha
An attempt to build the NimBLE_Client.ino sketch from NimBLE-1.4.1 'examples' area with arduino-cli for a ESP32-C6 devkit target gives a failure:
In file included from /home/codespace/.arduino15/packages/esp32/tools/esp32-arduino-libs/idf-release_v5.1-3662303f31/esp32c6/include/bt/include/esp32c6/include/esp_bt.h:17,
                 from /home/codespace/Arduino/libraries/NimBLE-Arduino-1.4.1/src/NimBLEDevice.h:46,
                 from /home/codespace/Arduino/libraries/NimBLE-Arduino-1.4.1/examples/NimBLE_Client/NimBLE_Client.ino:11:
/home/codespace/.arduino15/packages/esp32/tools/esp32-arduino-libs/idf-release_v5.1-3662303f31/esp32c6/include/bt/common/api/include/api/../../../../controller/esp32c6/esp_bt_cfg.h:19:10: fatal error: syscfg/syscfg.h: No such file or directory
   19 | #include "syscfg/syscfg.h"
      |          ^~~~~~~~~~~~~~~~~
compilation terminated.

I wish this were possible but it is sadly not. Support for C6 and H2 is hindered by the way the Arduino core is configured when the bluetooth libs are built as bluedroid is enabled there and the new IDF version uses the same symbols now for the controller interface and conflicts when linked.
The only way to support these 2 chips is either by using esp-idf and Arduino as a component (recommended) or using the Arduino lib builder to disable bluedroid and nimble, controller only config.

Hmm, i lied... kinda. Here is a branch that compiles, no idea if it will work, probably not due to mixing of functions but give it a go and let me know.
https://github.com/h2zero/NimBLE-Arduino/tree/esp32-c6-test

Thank you for development of the C6 branch !

branch that compiles

Compilation of NimBLE_Client.ino demo sketch goes fine.
However, the linker is still unable to complete it's task:
/home/codespace/.arduino15/packages/esp32/tools/esp-rv32/2302/bin/../lib/gcc/riscv32-esp-elf/12.2.0/../../../../riscv32-esp-elf/bin/ld: /home/codespace/.arduino15/packages/esp32/tools/esp32-arduino-libs/idf-release_v5.1-3662303f31/esp32c6/lib/libbt.a(os_msys_init.c.obj): in function `os_msys_init':
/home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/bt/porting/nimble/src/os_msys_init.c:90: multiple definition of `os_msys_init'; /tmp/arduino/libraries/NimBLE-Arduino/nimble/porting/nimble/src/os_msys_init.c.o:/home/codespace/Arduino/libraries/NimBLE-Arduino/src/nimble/porting/nimble/src/os_msys_init.c:190: first defined here
collect2: error: ld returned 1 exit status

Duplicate of os_msys_init symbol is detected.

That's odd, it seemed to link fine with the latest Arduino commit in platformio.

Arduino commit in platformio.

platformio typically lags few versions behind Espressif official Arduino Core.
The most recent one is 3.0.0-alpha3
To reproduce, please, use official Arduino CLI as follows
$ arduino-cli config init
$ arduino-cli config add board_manager.additional_urls https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_dev_index.json
$ arduino-cli core update-index
$ arduino-cli core install<EMAIL_ADDRESS>$ arduino-cli board attach -b esp32:esp32:esp32c6
$ arduino-cli compile -v  -b esp32:esp32:esp32c6 <path-to-Nim-BLE-examples>/NimBLE_Client/

Actually I modified platformio to be able to build from the latest commit on master branch. Thanks though, I'll try that and see if I can reproduce.

Reproduced and fixed, please do a reset --hard on the branch and try again. I'm not expecting it to work but worth a shot. Sadly I do not have one of these devices to test on.

Reproduced and fixed

Thank you so much !

I'm not expecting it to work but worth a shot.

Sure!
It enters into boot loop soon after Starting NimBLE Client message.
ESP-ROM:esp32c6-20220919
Build:Sep 19 2022
rst:0xc (SW_CPU),boot:0xc (SPI_FAST_FLASH_BOOT)
Saved PC:0x4001975a
SPIWP:0xee
mode:DIO, clock div:2
load:0x4086c410,len:0xc24
load:0x4086e610,len:0x2708
load:0x40875728,len:0x594
entry 0x4086c410
Starting NimBLE Client
Guru Meditation Error: Core  0 panic'ed (Load access fault). Exception was unhandled.

Will try to decode the backtrace later....

lol, not surprised at all. Mixing code is what's happening here. The real fix for this will be to rename A LOT of functions and hope that's enough.

I've just pushed an update that renames all the conflicting symbols. Please do a reset --hard on the branch and let me know what happens.

let me know what happens

A little bit different PC and SP, but the fault is of the same type
ESP-ROM:esp32c6-20220919
Build:Sep 19 2022
rst:0xc (SW_CPU),boot:0xc (SPI_FAST_FLASH_BOOT)
Saved PC:0x4001975a
SPIWP:0xee
mode:DIO, clock div:2
load:0x4086c410,len:0xc24
load:0x4086e610,len:0x2708
load:0x40875728,len:0x594
entry 0x4086c410
Starting NimBLE Client
Guru Meditation Error: Core  0 panic'ed (Store access fault). Exception was unhandled.

Thanks,that's too bad. Definitely need a backtrace to progress. I'll try to get my hands on a board to test with soon.

My ESP32-C6 board does not have USB/JTAG (GPIO12/13) pins exposed, so capturing a backtrace with assistance of OpenOCD will be problematic for a while...
I found out that ESP32-C3 build of the same sketch ( NimBLE_Client.ino ) with Arduino Core 3.0.0-alpha3 does also produce a binary that fails to boot properly.

So if you have ESP32-C3 readily available - you can try it first ( with 3.0.0-alpha3 ).

Interesting, thanks! Looks like the queue is not initialized for these boards... I'll look into that

@lyusupov,I just pushed a few commits to this, seems to work for me  😄

@h2zero
Thank you!
Yes, it looks good to me as well, both with ESP32-C3 and ESP32-C6:
ESP-ROM:esp32c6-20220919
Build:Sep 19 2022
rst:0x1 (POWERON),boot:0xc (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
mode:DIO, clock div:2
load:0x4086c410,len:0xc24
load:0x4086e610,len:0x2708
load:0x40875728,len:0x594
entry 0x4086c410
Starting NimBLE Client
controller lib commit: [77d09ce]
Advertised Device found: Name: , Address: xx:xx:xx:xx:xx:xx, manufacturer data:<PHONE_NUMBER>029cbc0cecd8caad59d727965b13de6546c8da802af2038f
Advertised Device found: Name: HONOR Band 5i-406, Address: xx:xx:xx:xx:xx:xx, manufacturer data: 7d02010300ffff, txPower: 4
                                                                                Service Data:
                                                                                UUID: 0x3802, Data: 8▒▒▒
Advertised Device found: Name: Galaxy Watch (CC1E) LE, Address: xx:xx:xx:xx:xx:xx, appearance: 192, manufacturer data:<PHONE_NUMBER>00010102

I suppose that your recent development efforts are sufficient to consider this GitHub ticket as 'complete'.
I am ready to close this issue report.
If me or someone else will face up with another ESP32-C6 NimBLE issue - we should open a new ticket.

Sounds good, thanks!
Going to need to test this for some time before merging into master.

Hello @h2zero , please accept my apology in advance for asking this in this thread, but I need some guidance/direction.
I have a DFRobot ESP32-C6 board, which I would like to use to send sensor data over BLE to a mobile application.
Let me give you a short background.. I do my hardest to learn and figure out things myself, (where possible), but Im certainly still a novice when it comes to coding.
I started in Arduino IDE with the IBeacon example sketch, got that working. My next step was to figure out how to send the sensor reading, instead of the "hello" over the broadcast (that how I ended up on this thread), and discovered NimBLE.
When running the NimBLE Client example, or any of the others, I get the "syscfg/syscfg.h" error, and that is how I ended up on this thread.
And this is where I feel I have gone down a very dark rabbit hole where I have installed VCCode and PlatformIO that you guys have been talking about in this thread and even now attempting the Platformio build that you and Jason from Tasmoto has been working on ..
As you can see, I might have complete gone off in the wrong direction.
If this is something that you could help me with, I would be grateful and if there is another way we could communicate, other than me spamming this thread, please let me know.
Thank you!!

Hello @NeilvdM, good for you for trying to sort this out and learn. You've gone in the right direction, it's just a little steeper of a learning curve 😄
The C6 is not able to be supported with this repo, but instead esp-nimble-cpp needs to be used, which means you will need to use esp-idf and Arduino as a component of that to get the Arduino interfaces. Or alternatively we can use a for of the Arduino core that @Jason2866 has created for use in platformio.
I would suggest starting with the core that tasmota uses: https://github.com/Jason2866/platform-espressif32 follow the readme to setup the environment.
Let me know how you get on with that and I'll try to help further.

You're getting close 🙂!
As for your question, yes you could use that ibeacon example and work from there, that will use the default BLE library in the official arduino core. This library is designed to be a lighter weight and more capable/stable replacement for that. What you should do depends on your needs.

obviously, the lighter weight and stability will be important for me .. so then what you are telling me is that I need to learn VSCode.
Then that brings me to these points:
[env:platformio]
platform = https://github.com/Jason2866/platform-espressif32.git#Arduino/IDF5
framework = arduino
board = esp32-c6-devkitc-1

is this correct?
Do I need to add your library now below the board?
LIB_DEPS =<EMAIL_ADDRESS>Then start playing with the example sketches?
Why when I copy an Example sketch from the NimBLE Github, for example, I get parsing errors in VSCode?
Would you be willing to share an email address that I can contact you on? (thats if you dont mind)
THANK YOU FOR THE INFO YOU HAVE SHARED SO FAR

You'll certainly need to learn to use VScode and platformio for this, you'll be glad you did later 😄
This is what you need:
[env:platformio]
platform = https://github.com/Jason2866/platform-espressif32.git#Arduino/IDF5
framework = arduino
board = esp32-c6-devkitc-1
lib_deps =<EMAIL_ADDRESS>
Then you can use the examples from here, you might have to add #include <Arduino.h> at the top of the file to get it to compile.
I prefer to keep the communication here so others can benefit.

Well I found the reason for my Parsing errors .. I was inserting the sketch in the .ini file, not the main.c :) .. lol
Now on to the next step .. I include the #include <Arduino.h> at the top line of main.c file, but I get the error
"message": "cannot open source file "Arduino.h"",
I followed an Indestructible example of getting an Arduino sketch to run in Platformio, the start of that .ini looks like:
[env:esp32dev]
platform = espressif32
board = esp32dev
framework = arduino

monitor_speed = 115200
board_build.partitions = huge_app.csv
lib_deps =
  https://github.com/trevorwslee/Arduino-DumbDisplay
<EMAIL_ADDRESS>build_flags =
  -DFOR_ESP32

That main.c also have #include <Arduino.h> as the first line, but that does not give the error
What am I missing?

ok, so to see if I can get the basics working, I configured my ini file as such:
[env:platformio]
platform = https://github.com/Jason2866/platform-espressif32.git#Arduino/IDF5
framework = arduino
board = esp32-c6-devkitc-1
lib_deps =<EMAIL_ADDRESS>
Then in the main.ccp I followed your beginner example to create a BLE server
#include NimBLEDevice.h

// void setup() in Arduino
void app_main(void)
{
    NimBLEDevice::init("NimBLE");

    NimBLEServer *pServer = NimBLEDevice::createServer();
    NimBLEService *pService = pServer->createService("ABCD");
    NimBLECharacteristic *pCharacteristic = pService->createCharacteristic("1234");

    pService->start();
    pCharacteristic->setValue("Hello BLE");

    NimBLEAdvertising *pAdvertising = NimBLEDevice::getAdvertising();
    pAdvertising->addServiceUUID("ABCD");
    pAdvertising->start();
}

But this is the error that Im getting now:

Processing * in esp32-c6-devkitc-1 environment
Building & Uploading...
Library Manager: Installing throwtheswitch/Unity @ ^2.5.2
Unpacking  [####################################]  100%
Library Manager<EMAIL_ADDRESS>has been installed!
Error: Nothing to build. Please put your test suites to the 'D:\OneDrive\Personal\Documents\PlatformIO\Projects\ESP32C-6\test' folder
Uploading stage has failed, see errors above. Use pio test -vvv option to enable verbose output.
--------------------------------------------------------------------- esp32-c6-devkitc-1:* [ERRORED] Took 13.14 seconds ------------
======================= SUMMARY
Environment         Test    Status    Duration

esp32-c6-devkitc-1  *       ERRORED   00:00:13.145

At this rate, it feels like Im never gonna get any where

@NeilvdM I suggest you consider the pioarduino fork which I have nicely working with NImble-Arduino master here
start here: https://github.com/espressif/arduino-esp32/discussions/10039 - maybe check the discord channel
to here: https://github.com/pioarduino/platform-espressif32

@NeilvdM I suggest you consider the pioarduino fork which I have nicely working with NImble-Arduino master here

you are joking with this right? :) I just opened that page and its like Senior Varsity stuff, that is like throwing me at the bottom of the Mariana Trench and say, "alright, there you go, swim to the top" .. lol
There is so much information there, I dont know where to look at.
But In any case, Ill check out the other links you suggested.
Thanks

@NeilvdM Sorry for the delay, it looks like you have some issue with the folder arrangement, I will email you.

I made an example. It is for the esp32 for the esp32-c6 it should work too. With only replacing the board against a c6 one.
https://github.com/pioarduino/platform-espressif32/tree/develop/examples/espidf-arduino-h2zero-BLE_scan

I've re-opened this ticket due to high message traffic.
@h2zero please, feel yourself free to close it again when it is done.

I would also love to have a working solution for this. +1

i am sorry but I am a little bit confused. Is it a chance now to use nimBLE with arduino framework and esp-c6? (not arduino as an idf-component). I've tried to compile the release 1.4.2 , using: platform = https://github.com/pioarduino/platform-espressif32/releases/download/51.03.04/platform-espressif32.zip
With these configuration , it doesn't compile, but @mhaberler mentioned he succeded in.

Currently working on making it easier to compile Arduino projects with IDF using Platformio.
With a slightly modified IDF (just warning settings changed) project Tasmota compiled without changing a single line of code with Arduino/espidf.

It is not currently possible to use the C6 with this library and Arduino core.

The same is for the ESP32-H2, and the Android 3.1.0RC1 :pleading_face:

Where did the branch https://github.com/h2zero/NimBLE-Arduino#esp32-c6-test go?

Hello @avaksru that branch was just for testing and has been deleted. I've aggregated the changes from the test branches into 1 here: https://github.com/h2zero/NimBLE-Arduino/tree/c6-build
Not sure what you mean by everything worked? The only thing that is working is scanning.

Thanks for the quick response. The scan was working. Everything was assembled from the #esp32-c6- test branch without errors. And it worked great! But from the #c6-build branch, the build goes through with errors.
.platformio/packages/toolchain-riscv32-esp/bin/../lib/gcc/riscv32-esp-elf/12.2.0/../../../../riscv32-esp-elf/bin/ld.exe: .pio/build/esp32c6_4mb/libb1c/libNimBLE-Arduino.a(ble_hs.c.o): in function ble_hs_tx_data':
.pio/libdeps/esp32c6_4mb/NimBLE-Arduino/src/nimble/nimble/host/src/ble_hs.c:584: undefined reference to `ble_hci_trans_funcs_ptr'
.platformio/packages/toolchain-riscv32-esp/bin/../lib/gcc/riscv32-esp-elf/12.2.0/../../../../riscv32-esp-elf/bin/ld.exe: .pio/libdeps/esp32c6_4mb/NimBLE-Arduino/src/nimble/nimble/host/src/ble_hs.c:731: undefined reference to `ble_hci_trans_funcs_ptr'
.platformio/packages/toolchain-riscv32-esp/bin/../lib/gcc/riscv32-esp-elf/12.2.0/../../../../riscv32-esp-elf/bin/ld.exe: .pio/build/esp32c6_4mb/libb1c/libNimBLE-Arduino.a(ble_hs_flow.c.o): in function ble_hs_flow_startup': .pio/libdeps/esp32c6_4mb/NimBLE-Arduino/src/nimble/nimble/host/src/ble_hs_flow.c:260: undefined reference to ble_hci_trans_funcs_ptr'
....`

It builds fine with:
  framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#3.0.1
  framework-arduinoespressif32-libs @ https://github.com/espressif/arduino-esp32/releases/download/3.0.1/esp32-arduino-libs-3.0.1.zip

Keep in mind that this is just a test area and that this branch is not supported for use.

@avaksru Fully working is using h2zero IDF NimBLE and compile it with Arduino as a component of IDF. See this example https://github.com/pioarduino/platform-espressif32/tree/Arduino/IDF53/examples/espidf-arduino-h2zero-BLE_scan
To use change in the example platformio.ini the platform entry to:
platform = https://github.com/pioarduino/platform-espressif32.git#Arduino/IDF53
Mandantory the idf_component.yml!

Same as @avaksru with arduino-esp32 v3.1.0
Reverting to 3.0.1 works fine tho
@Jason2866 the link https://github.com/pioarduino/platform-espressif32/tree/Arduino/IDF53/examples/espidf-arduino-h2zero-BLE_scan is dead

@pierreyves258  Yes, it is merged in branch master https://github.com/pioarduino/platform-espressif32/tree/main/examples/espidf-arduino-h2zero-BLE_scan


authentication policy side-effects and debugging

I have a service that I want to protect with a JWT. So I create an authentication policy looks like this:
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
  name: auth-spec
  namespace: default
spec:
  origins:
  - jwt:
      issuer: https://apigee-cps-test.apigee.net/istio-auth/token
      jwksUri: https://apigee-cps-test.apigee.net/istio-auth/certs
  principalBinding: USE_ORIGIN
  targets:
  - name: helloworld

And this worked great! However, I gave it to another user who applied it to their mesh and they started getting a 503 error with this message when curling to their service:
upstream connect error or disconnect/reset before headers

Not much to go on there, so they asked for my help and I started poking around. Unfortunately, I didn't find anything useful in the logs, they were all similar to this:
[2018-07-27T17:36:34.765Z] "GET /hello HTTP/1.1" 503 UF 0 57 3 - "<IP_ADDRESS>" "curl/7.54.0" "d159de85-7442-969c-a440-2806d61da2b3" "<IP_ADDRESS>" "<IP_ADDRESS>:5000"

In other words, they reflected what I already knew at the client: The system was returning a 503.
So, I moved to other debugging techniques. After a while, I found that if I curled the pod container from inside the mesh without the auth header, I’d get the normally expected 401 error. Aha! So the proxy was doing it's job. On the other hand, if I curled the service from inside the mesh, I’d get the 503 error. Well, at least now I knew that something was happening between the service and the pod. But what?
After more futzing around and pouring over docs and such, I finally tried changing my auth policy. I added this little gem to policy above and applied it:
peers:

mtls:

And presto! Things started working! Hmm...
So, the intent to add JWT verification actually overrode the ability of the service to work in the mTLS environment. Ouch. That was an unexpected side-effect - there was nothing in my policy that would have indicated that. (On the contrary, it was the omission of this section that caused the service behavior to be overridden contrary to the existing behavior and the rest of the Istio system.)
Anyway, to sum-up my concerns:

there are potentially hidden side-effects to the authentication policy
if you run into issues, it's difficult to debug as neither the error message nor the logs help

And my desired behavior is:

Separate concerns so there aren't side-effects like this
Improve errors and/or logging so that if there are reachability issue, it's obvious why

I hope that all makes sense! Thanks!

Hi,
Thank you for sharing your experience. There are still a lot of works to improve usability. We have few standing issues that try to cover some of your suggestion, e.g this and this for better authentication failure message. Hope we will be able to handle them in the near future.
We are also updating the documents for 1.0 release. Please check out the new authentication policy task, specifically the End-user authentication section that describes exactly this situation.


doc: Sync classref with current source

Includes various changes triggered by the refactoring of method bindings in #42780.

Draft as some of the changes might be bugs that need to be addressed before we make them harder to spot.

Updated with classref changes from #43297 (+ added missing bindings).

I opened #43311 to keep track of the lost method bindings.


async_hooks.currentId is not a function

While trying to debug #51, I tried turning on the DEBUG_CLS_HOOKED environment variable, and it pretty quickly crashed.  Looks like this line:
https://github.com/Jeff-Lewis/cls-hooked/blob/master/context.js#L52
was meant to call executionAsyncId instead?  If you can confirm that this is the case, I'm happy to submit a pull request!

@Quantumplation at quick glance it looks like you made a good catch. I have to confirm but I think the newer versions of node, the currentID() has been removed. This should only be an issue when setting the DEBUG_CLS_HOOKED variable or if Babel is trying to evaluate all code. I'll take a closer look.


Support limiting the number of jobs returned by list-job

Right now list-job always returns the full list of all jobs throughout history (possibly limited to a specific pipeline, and recently limited to jobs from the most recent pipeline version by default). For the dash, however, we generally only need details about the most recent few jobs. @msteffen suggested that, though it may be difficult to get only the most recent few jobs from etcd, it would certainly be possible to limit the number of jobs for which we make the slow queries to object storage to populate all details.
I think this is particularly important because of the issue identified in #2791
This is basically a smaller ask, but related to general pagination requested in #2938

list job is designed with this in mind, it returns the most recent jobs first and you can hangup at any time when you've read as much as you want. That being said it would still probably make sense to add flags for explicitly controlling the pagination. And ideally have the same set of flags present in all of our streaming commands.


How to debug backup solution with amazon s3

Type of question
how to implement a specific feature
Question
What did you do?
cassandra-backup.yml
apiVersion: db.orange.com/v2
kind: CassandraBackup
metadata:
  name: daily-cassandra-backup
  namespace: cassandra
  labels:
    app: cassandra
spec:
  cassandraCluster: cassandra
  datacenter: datacenter1
  storageLocation: s3://backups-dev
  snapshotTag: daily
  secret: cassandra-dev-s3-backup-secret
  schedule: "@daily"

cassandra-dev-s3-backup-secret.yml:
apiVersion: v1
kind: Secret
metadata:
  name: cassandra-dev-s3-backup-secret
  namespace: cassandra
type: Opaque
stringData:
  awsaccesskeyid: <keyid>
  awssecretaccesskey: <secret>
  awsregion: eu-central-1
  awsendpoint: https://s3.eu-central-1.amazonaws.com

What did you expect to see?
I see this in CassandraBackup CRD with all the Annotations, but I don't know how to verify if it actually works. No file is created in AWS bucket.
What did you see instead? Under which circumstances?
No data in AWS, no events for created backup CRD
Environment

casskop version:

2.1.17

Kubernetes version information:

Kustomize Version: v5.0.1
Server Version: v1.25.9+k3s1

Kubernetes cluster kind:

k3s - 3 master 6 workers

Cassandra version:

3.11.3
replicas: 3

I got it working by changing name of cassandra cluster and the events show properly, but backup is still having problems cause of not specified --data-dir:
12:33:05.670 ERROR com.instaclustr.operations.Operation - Reporting errors for operation d3f8f0a3-13b8-4b4f-915f-934955e5e18b
12:33:05.670 ERROR com.instaclustr.operations.Operation - --- start of error reporting ---
12:33:05.670 ERROR com.instaclustr.operations.Operation - Error{source=cassandra-opt-datacenter1-rack1-1, message=You have not specified any --data-dir on command line or respective field in JSON request., throwable=java.lang.IllegalStateException: You have not specified any --data-dir on command line or respective field in JSON request.}
java.lang.IllegalStateException: You have not specified any --data-dir on command line or respective field in JSON request.
	at com.instaclustr.esop.impl.backup.BackupOperationRequest.validate(BackupOperationRequest.java:177)
	at com.instaclustr.esop.impl.backup.BackupOperation.run0(BackupOperation.java:131)
	at com.instaclustr.operations.Operation.run(Operation.java:247)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
12:33:05.674 ERROR com.instaclustr.operations.Operation - --- end of error reporting ---

How to fix this?

I got it working by changing name of cassandra cluster and the events show properly, but backup is still having problems cause of not specified --data-dir:
12:33:05.670 ERROR com.instaclustr.operations.Operation - Reporting errors for operation d3f8f0a3-13b8-4b4f-915f-934955e5e18b
12:33:05.670 ERROR com.instaclustr.operations.Operation - --- start of error reporting ---
12:33:05.670 ERROR com.instaclustr.operations.Operation - Error{source=cassandra-opt-datacenter1-rack1-1, message=You have not specified any --data-dir on command line or respective field in JSON request., throwable=java.lang.IllegalStateException: You have not specified any --data-dir on command line or respective field in JSON request.}
java.lang.IllegalStateException: You have not specified any --data-dir on command line or respective field in JSON request.
	at com.instaclustr.esop.impl.backup.BackupOperationRequest.validate(BackupOperationRequest.java:177)
	at com.instaclustr.esop.impl.backup.BackupOperation.run0(BackupOperation.java:131)
	at com.instaclustr.operations.Operation.run(Operation.java:247)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
12:33:05.674 ERROR com.instaclustr.operations.Operation - --- end of error reporting ---

How to fix this?

Sorry @mac-lucky I've been away for some time. Looking into it.. Any information in the events besides those logs ?
@AKamyshnikova do you think it's been caused by the upgrade then downgrade ? Do you have the same issue ?
I see that we haven't created a version containing
* [e2e tests] Fix api version for casskop custom resources by @AKamyshnikova in https://github.com/cscetbon/casskop/pull/119
* Revert "Update instaclustr-icarus to 2.0.4" by @AKamyshnikova in https://github.com/cscetbon/casskop/pull/121

and I wonder if this is why.

@cscetbon I hoped we merge https://github.com/cscetbon/casskop/pull/120 and create new release

In events i got
Backup operation 8bf7508b-242b-4a5a-bc7a-0d2af4c5f1d7 on node cassandra-opt-datacenter1-rack1-1 has failed
and the logs again from backrest-sidecar:
10:24:14.318 ERROR com.instaclustr.operations.Operation - Error{source=cassandra-opt-datacenter1-rack1-0, message=You have not specified any --data-dir on command line or respective field in JSON request., throwable=java.lang.IllegalStateException: You have not specified any --data-dir on command line or respective field in JSON request.}
java.lang.IllegalStateException: You have not specified any --data-dir on command line or respective field in JSON request.
	at com.instaclustr.esop.impl.backup.BackupOperationRequest.validate(BackupOperationRequest.java:177)
	at com.instaclustr.esop.impl.backup.BackupOperation.run0(BackupOperation.java:131)
	at com.instaclustr.operations.Operation.run(Operation.java:247)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
10:24:14.318 ERROR com.instaclustr.operations.Operation - --- end of error reporting ---

Yeah @mac-lucky that makes sense and seems related to what @AKamyshnikova was talking about. We need to create a new release and will do soon

I see there is new version released, but I don't see any new helm chart? Is there some CI/CD for this or You build this manually? :)

Everything is functioning well now.

@mac-lucky glad to hear !

When a backup is created daily, does it have a retention period or is it overwritten each day? Based on my observation, it seems to be the latter option. Could you please confirm this?

So backup are not actually cleaned. Files are not overwritten if they still exist when the next backup happens but their metadata is updated. That way if you use AWS S3 Lifecycle for instance to clean up older files, those files won't be deleted as they were just updated (only their metadata) recently. To read more about it checkout esop's documentation as it uses Icarus under the hood which is what CassKop uses as well

@cscetbon Thank you, I read it. So if I would like to have a backup from 7 days I need 7 buckets and 7 jobs with backup?

Fortunately not ! You just take a daily backup (for instance by setting schedule: "@midnight" in your CassandraBackup object) and create an s3 lifecycle policy that gets rid of files that are older than 7 days (check out this blog post)

Last question. How do I recover backup from 3 days ago?  Can I specify backup timestamp?
apiVersion: db.orange.com/v2
kind: CassandraRestore
metadata:
  name: nightly-cassandra-backup
  labels:
    app: cassandra
spec:
  cassandraBackup: nightly-cassandra-backup
  cassandraCluster: test-cluster

You you can't recover a backup to a specific date but only to the last version of it. If you want to be able to choose which day to pick, then you need one CassandraBackup per day instead.
So you could have one for Monday, one for Tuesday, one for Wednesday, etc... each one scheduled on a specific day only and then you can choose the exact one you want to restore.

Closing this request as questions got answered


[BACKLOG-19195] Added SSL support to the http client manager.

Please merge along with: https://github.com/pentaho/pentaho-ee/pull/701

Build Completed
:x: This pull request has errors. They will need to be addressed before it can be accepted. See below for more details. Some links are also available below for further assistance in addressing those issues.
Build Commands
mvn -B -fn -f 'pom.xml' -pl 'core' -P '!assemblies' -amd clean install && mvn -B -f 'pom.xml' -pl 'core' -P '!assemblies' -amd site

Cleanup Commands
mvn -B -f 'pom.xml' -pl 'core' -P '!assemblies' -amd build-helper:remove-project-artifact

Changed files
  core/src/main/java/org/pentaho/di/core/util/HttpClientManager.java

Unit test coverage change
These statistics help you identify how your changes have affected the coverage of the following files. If a file is not in this list, then its coverage was not affected by your changes. To get some help interpreting these metrics, please refer to Jacoco's documentation.
org.pentaho.di.job.entries.evalfilesmetrics.JobEntryEvalFilesMetrics

Branch Change: -.38%:small_red_triangle_down:
Complexity Change: -.51%:small_red_triangle_down:
Instruction Change: -.08%:small_red_triangle_down:

org.pentaho.di.job.entries.evaluatetablecontent.JobEntryEvalTableContent

Branch Change:  + 1.01%
Complexity Change:  + 1.06%
Instruction Change:  + .17%

org.pentaho.di.job.entries.sql.JobEntrySQL

Instruction Change:  + 3.28%
Line Change:  + 3.87%

org.pentaho.di.trans.step.StepMeta

Branch Change: -.68%:small_red_triangle_down:
Complexity Change: -.58%:small_red_triangle_down:
Instruction Change: -.19%:small_red_triangle_down:
Line Change: -.26%:small_red_triangle_down:

org.pentaho.di.trans.steps.combinationlookup.CombinationLookupMeta

Instruction Change: -.04%:small_red_triangle_down:

org.pentaho.di.trans.steps.constant.ConstantMeta

Instruction Change:  + .10%

org.pentaho.di.trans.steps.csvinput.CsvInputMeta

Instruction Change:  + .05%

org.pentaho.di.trans.steps.datagrid.DataGridMeta

Branch Change: -1.92%:small_red_triangle_down:
Complexity Change: -1.69%:small_red_triangle_down:
Instruction Change: -.17%:small_red_triangle_down:

org.pentaho.di.trans.steps.dbproc.DBProcMeta

Instruction Change: -.11%:small_red_triangle_down:

org.pentaho.di.trans.steps.dimensionlookup.DimensionLookupMeta

Branch Change:  + 1.22%
Complexity Change:  + .76%
Instruction Change:  + .29%
Method Change: -1.01%:small_red_triangle_down:

org.pentaho.di.trans.steps.groupby.GroupByMeta

Branch Change: -3.12%:small_red_triangle_down:
Complexity Change: -3.16%:small_red_triangle_down:
Instruction Change: -.28%:small_red_triangle_down:
Line Change: -.74%:small_red_triangle_down:

org.pentaho.di.trans.steps.ifnull.IfNullMeta

Branch Change:  + 2.00%
Complexity Change:  + 1.89%
Instruction Change:  + .17%

org.pentaho.di.trans.steps.insertupdate.InsertUpdateMeta

Instruction Change: -.05%:small_red_triangle_down:

org.pentaho.di.trans.steps.memgroupby.MemoryGroupByMeta

Branch Change: -3.37%:small_red_triangle_down:
Complexity Change: -3.90%:small_red_triangle_down:

org.pentaho.di.trans.steps.setvalueconstant.SetValueConstantMeta

Instruction Change:  + .14%

org.pentaho.di.trans.steps.singlethreader.SingleThreaderMeta

Instruction Change: -.10%:small_red_triangle_down:

org.pentaho.di.trans.steps.sortedmerge.SortedMergeMeta

Instruction Change:  + .17%

org.pentaho.di.trans.steps.webservices.WebServiceMeta

Branch Change:  + 1.79%
Complexity Change:  + 1.19%
Instruction Change:  + .07%

org.pentaho.di.www.TransformationMap

Branch Change:  + 4.55%
Complexity Change:  + 3.08%
Instruction Change:  + 3.92%
Line Change:  + 4.62%

org.pentaho.di.trans.steps.googleanalytics.GaInputStepMeta

Branch Change:  + 1.85%
Complexity Change:  + 1.06%
Instruction Change:  + .13%
Line Change:  + .33%

org.pentaho.di.trans.steps.salesforceupdate.SalesforceUpdateMeta

Instruction Change:  + .17%

Checkstyle violations
The code submitted is not complying with one or more of our guidelines. To perform these checks locally, please use this tool.
/home/buildguy/project/core/src/main/java/org/pentaho/di/core/util/HttpClientManager.java

:heavy_exclamation_mark: ',' is preceded with whitespace. at Line: 62 Column: 30

:heavy_exclamation_mark: ',' is preceded with whitespace. at Line: 63 Column: 31


Feature request: nested record access

For the AGREE tool we are generating Lustre with nested records. Based on kind2 outout and browsing the Lustre grammar, it looks like kind2 does not yet supported nested record access. For example, the following file generates a parser error:
type point = struct { x : int; y : int };
type weighted_point = struct { weight : real; p : point };

node main(wp : weighted_point) returns (sum : int);
let
  sum = wp.p.x + wp.p.y;
tel;

Yields
Parser error in line 6 col. 12: Syntax error

John Backes said this issue has been fixed. Is that true?

Version 0.8 does not support it, but the contracts and arrays branch (which will soon become the official develop branch) does.

The contracts and arrays branch has become the official develop branch, closing issue.


integrate building flists in github workflow

check https://github.com/threefoldtech/home/issues/718#issuecomment-635707684
we can use the playground to push to the hub with guest credentials

integrated in the pipeline https://github.com/threefoldtech/js-ng/blob/development/.github/workflows/build_tagged_image.yml#L24


fix #500: Add language parameter to GetTvShowVideosAsync method

fix #500 - The GetTvShowVideosAsync method has been updated to include an optional language parameter. This parameter allows users of the TMDbClient library to retrieve videos in a specific language instead of only in default(en-US)

any update on it being merged?


Capstone Project Module 2

https://github.com/oovillagran/Capstone-Module2/projects/1
There are two people in our team:

@naanmohammed
@oovillagran

STATUS: CHANGES REQUIRED ♻
Hi @oovillagran, and @naanmohammed👋,
Good job so far!
There are some issues that you still need to work on to go to the next project but you are almost there!

Required Changes ♻

[ ] Please make sure to assign team members to all group task cards as you did for others. ( You can divide the group tasks among team members or you can assign all team members to group tasks).

Optional suggestions
Every comment with the [OPTIONAL] prefix is not crucial enough to stop the approval of this PR. However, I strongly recommend you to take them into account as they can make your code better.
You can also consider:
N/A
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Please, remember to tag me in your question so I can receive the notification.
Please, do not open a new Pull Request for re-reviews. You should use the same Pull Request submitted for the first review, either valid or invalid unless it is requested otherwise.

As described in the Code reviews limits policy you have a limited number of reviews per project (check the exact number in your Dashboard). If you think that the code review was not fair, you can request a second opinion using this form.

Hello @thecodechaser,
Thank you for your kind review. We already assign all the activities as required.👍

Approved 🥳
Hi @oovillagran & @naanmohammed 👋
Great work implementing the suggested changes. 💪
Your project is COMPLETE! There is nothing else to say other than... it's time to merge it! 🚀 Congratulations!

Highlights 🎯
There are some excellent points that you have made perfectly. Keep up the great work! 👏

✅ All cards are assigned to team members
✅ Cards are assigned in a fair way
✅ Cards with group task are assigned

Optional suggestions ❕
Every comment with the [OPTIONAL] prefix is not crucial enough to stop the approval of this PR. However, I strongly recommend you to take them into account as they can make your code better.
N/A
Cheers and Happy coding!👏👨💻💻
Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Please, remember to tag me in your question so I can receive the notification.
Please, do not open a new Pull Request for re-reviews. You should use the same Pull Request submitted for the first review, either valid or invalid unless it is requested otherwise.

As described in the Code reviews limits policy you have a limited number of reviews per project (check the exact number in your Dashboard). If you think that the code review was not fair, you can request a second opinion using this form.


[docs] rule staging and promo documentation

to: @austinbyers
cc: @airbnb/streamalert-maintainers
size: medium
Background
Context for the change
Changes

Renaming rule-table subcommand to rule-staging

Adding new command for enabling/disabling rule staging feature

Adding documentation for the rule staging and rule promotion features.

Testing
Built and tested docs locally. I advised doing the same when reviewing :)

Coverage remained the same at 97.234% when pulling 643223df6a8fb9e59a2408a62ff738fe647e8b85 on ryandeivert-rule-stage-promo-docs into a7eaa30516856e17303244fc99cb386be697e69f on master.

@austinbyers I added one commit with more info - PTAL


Add stale bot

We use the same at Guzzle.
Needs to be configured from: https://github.com/apps/stale

hey @colinodell do you have admin rights for the League Org to add StaleBot to this repo?

Yep, I do have admin rights, and it looks like it's currently enabled for all repositories in the org :) Feel free to ping me if you run into any issues or if I can assist further.

fantastic, thank you so much @colinodell!

Thank you for merging.
I created some labels. Lets see if the bot is working. Im not sure if it is supposed to take a day or two


_CastError (Null check operator used on a null value)

Hi @MarcelGarus ,
Thank you for this wonderful library, as it is part of my project.
But I have some hiccups while  using this library, and sometimes it's crashing by throwing the below error.
_CastError (Null check operator used on a null value)

I am using the latest version of marquee (2.2.3), and below are my flutter version details.
    • Flutter version 3.0.5 at C:\Users\dines\workspace\flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision f1875d570e (6 weeks ago), 2022-07-13 11:24:16 -0700
    • Engine revision e85ea0e79c
    • Dart version 2.17.6
    • DevTools version 2.12.2

And I am using marquee as below, which is causing the exception

Container(
            decoration: BoxDecoration(
              color: Theme.of(context).colorScheme.primaryContainer,
            ),
            height: Adaptive.h(4),

            child: Marquee(
              text: announce.announce,
              style: Theme.of(context).textTheme.bodyLarge!.copyWith(
                  color: Theme.of(context).colorScheme.onPrimaryContainer),
              scrollAxis: Axis.horizontal,
              crossAxisAlignment: CrossAxisAlignment.start,
              blankSpace: Adaptive.w(50),
              velocity: 10.0,
              numberOfRounds: null,
              pauseAfterRound: Duration(seconds: 1),
              startPadding: 10.0,
              accelerationDuration: Duration(seconds: 1),
              accelerationCurve: Curves.easeIn,
              decelerationDuration: Duration(milliseconds: 500),
              decelerationCurve: Curves.easeOut,
            ),
          )

This actually happens, while scrolling the Listview  where Marquee is one of it's children.
Please let me know if I miss anything here.

I have the same problems:(

@MarcelGarus Same issue here also


Nvidia Jetson TX2  Check failed: (err) == (cudaSuccess) Name: mxnet_generic_kernel ErrStr:no kernel image is available for execution on the device

Description
Can't use GPU on TX2. I compile mxnet by myself(http://mxnet.incubator.apache.org/install/index.html
Linux-Python-GPU-Build from source),and there is no error while compiling.   When I test my mxnet, there is  an error.

Check failed: (err) == (cudaSuccess) Name: mxnet_generic_kernel ErrStr:no kernel image is available for execution on the device

Environment info (Required)
TX2
CUDA 9.0 cuDNN 7.0
mxnet->master

I have solved this problem. The default CUDA_ARCH doesn't contain the TX2 's CUDA_ARCH. So we need to add "CUDA_ARCH = -gencode arch=compute_62,code=sm_62" to make/config.mk manually.
(https://discuss.gluon.ai/t/topic/6589)


Move CI jobs to .prow.yaml

What this PR does / why we need it:
This PR moves all presubmits and postsubmits to the .prow.yaml file.
Does this PR introduce a user-facing change?:
NONE

/assign @kron4eg

It's expected that pull-kubermatic-kubeone-validate-prow-yaml will fail until infra PR is not merged.

/retest


Aborting on pipeline job failure

This issue could potentially be split into two separate ones.  I have run a number of workflows which appear to succeed when viewing the graph in the Dataflow UI, but on closer inspection of the logs for the individual GGP jobs it turns out that some of the individual steps are not actually succeeding.
Here is the beginning part of a representative "task.log" file produced by one of the failed jobs:
2016/09/29 21:49:13 I: Switching to status: pulling-image
2016/09/29 21:49:13 I: Pulling image "b.gcr.io/isb-cgc-public-docker-images/gdc-client:1.0.1"
2016/09/29 21:49:23 I: Pulled image "b.gcr.io/isb-cgc-public-docker-images/gdc-client:1.0.1" successfully.
2016/09/29 21:49:23 I: Switching to status: localizing-files
2016/09/29 21:49:23 I: Docker file /gdc-data/418122394-gdc-user-token.2016-09-29T14_45_50-07_00.txt maps to host location /mnt/gdc-data/418122394-gdc-user-token.2016-09-29T14_45_50-07_00.txt.
2016/09/29 21:49:23 I: Copying <REDACTED> to /mnt/gdc-data/418122394-gdc-user-token.2016-09-29T14_45_50-07_00.txt
2016/09/29 21:49:23 I: Running command: sudo gsutil -q -m cp <REDACTED> /mnt/gdc-data/418122394-gdc-user-token.2016-09-29T14_45_50-07_00.txt
2016/09/29 21:49:40 I: Done copying files.
2016/09/29 21:49:40 I: Switching to status: running-docker
2016/09/29 21:49:40 I: Setting these data volumes on the docker container: [-v /tmp/ggp-160996381:/tmp/ggp-160996381 -v /mnt/gdc-data:/gdc-data]
2016/09/29 21:49:40 I: Running command: docker run -v /tmp/ggp-160996381:/tmp/ggp-160996381 -v /mnt/gdc-data:/gdc-data -e UUID=0001801b-54b0-4551-8d7a-d66fb59429bf -e DISK_SIZE=100 -e GDC_TOKEN=/gdc-data/418122394-gdc-user-token.2016-09-29T14_45_50-07_00.txt -e DOWNLOADED=/gdc-data/1430729090-outputs b.gcr.io/isb-cgc-public-docker-images/gdc-client:1.0.1 /tmp/ggp-160996381
2016/09/29 21:54:13 I: Copying /var/log/google-genomics/*.log to gs://germline-testing/dockerflow/logs/gdcDl/
2016/09/29 21:54:13 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://germline-testing/dockerflow/logs/gdcDl/
2016/09/29 21:59:13 I: Copying /var/log/google-genomics/*.log to gs://germline-testing/dockerflow/logs/gdcDl/
2016/09/29 21:59:13 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://germline-testing/dockerflow/logs/gdcDl/
2016/09/29 22:04:13 I: Copying /var/log/google-genomics/*.log to gs://germline-testing/dockerflow/logs/gdcDl/
2016/09/29 22:04:13 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://germline-testing/dockerflow/logs/gdcDl/
2016/09/29 22:04:44 I: Switching to status: delocalizing-files
2016/09/29 22:04:45 I: Docker file /gdc-data/1430729090-outputs maps to host location /mnt/gdc-data/1430729090-outputs.
2016/09/29 22:04:45 I: Copying /mnt/gdc-data/1430729090-outputs to gs://germline-testing/outputs
2016/09/29 22:04:45 I: Running command: sudo gsutil -q -m cp -L /var/log/google-genomics/out.log /mnt/gdc-data/1430729090-outputs gs://germline-testing/outputs
2016/09/29 22:04:45 E: command failed: CommandException: No URLs matched: /mnt/gdc-data/1430729090-outputs
CommandException: 1 file/object could not be transferred.
 (exit status 1)

One thing that I notice in this log file is that it looks like the outputs are expected to be available in a directory with a name following the scheme <numerical_string>-outputs that is supposedly located in at the root of the attached disk once the user specified command completes, but this is not how I've specified the outputs myself so I'm not sure where that's coming from.  I've been putting all of my inputs at the root of the attached disk and specifying individual outputs to be collected from there as well.
As for aborting the Dataflow workflow on failure, I found this comment in the WaitForOperation class:
// Check for errors and abort if that's the policy
which suggests that there might be a command line flag that could be used to enable this functionality.  Is this the case, or is there some other mechanism used to set the policy on aborting on failure?

First, on the pipeline not failing:
--abort=true should cause errors to exit. I thought that was the default.
You can explicitly set it if that's not the case -- and you should see it
in stdout if you use DirectPipelineRunner.
For file copies, the goal is that you never need to set local paths
explicitly. Set the GCS path, and then refer to the path on the VM as
${OUTPUT_FILE} or ${INPUT_FILE}, e.g.
In practice, the way it works is that the GCS path is hashed (the random
number you see. That flattens the folder hierarchy, avoids overwrites, and
makes sure files in the same folder remain in the same sort order.
Note that Dockerflow does not support recursive copy or folder copies. You
have to list files individually or use a wildcard, just as with Pipelines
API directly.
Can you share the full command-line and logs? (By private email is fine.)
On Thu, Sep 29, 2016 at 3:39 PM Abigail Hahn<EMAIL_ADDRESS>wrote:

This issue could potentially be split into two separate ones. I have run a
number of workflows which appear to succeed when viewing the graph in the
Dataflow UI, but on closer inspection of the logs for the individual GGP
jobs it turns out that some of the individual steps are not actually
succeeding.
Here is the beginning part of a representative "task.log" file produced by
one of the failed jobs:
2016/09/29 21:49:13 I: Switching to status: pulling-image
2016/09/29 21:49:13 I: Pulling image "b.gcr.io/isb-cgc-public-docker-images/gdc-client:1.0.1"
2016/09/29 21:49:23 I: Pulled image "b.gcr.io/isb-cgc-public-docker-images/gdc-client:1.0.1" successfully.
2016/09/29 21:49:23 I: Switching to status: localizing-files
2016/09/29 21:49:23 I: Docker file /gdc-data/418122394-gdc-user-token.2016-09-29T14_45_50-07_00.txt maps to host location /mnt/gdc-data/418122394-gdc-user-token.2016-09-29T14_45_50-07_00.txt.
2016/09/29 21:49:23 I: Copying  to /mnt/gdc-data/418122394-gdc-user-token.2016-09-29T14_45_50-07_00.txt
2016/09/29 21:49:23 I: Running command: sudo gsutil -q -m cp  /mnt/gdc-data/418122394-gdc-user-token.2016-09-29T14_45_50-07_00.txt
2016/09/29 21:49:40 I: Done copying files.
2016/09/29 21:49:40 I: Switching to status: running-docker
2016/09/29 21:49:40 I: Setting these data volumes on the docker container: [-v /tmp/ggp-160996381:/tmp/ggp-160996381 -v /mnt/gdc-data:/gdc-data]
2016/09/29 21:49:40 I: Running command: docker run -v /tmp/ggp-160996381:/tmp/ggp-160996381 -v /mnt/gdc-data:/gdc-data -e UUID=0001801b-54b0-4551-8d7a-d66fb59429bf -e DISK_SIZE=100 -e GDC_TOKEN=/gdc-data/418122394-gdc-user-token.2016-09-29T14_45_50-07_00.txt -e DOWNLOADED=/gdc-data/1430729090-outputs b.gcr.io/isb-cgc-public-docker-images/gdc-client:1.0.1 /tmp/ggp-160996381
2016/09/29 21:54:13 I: Copying /var/log/google-genomics/.log to gs://germline-testing/dockerflow/logs/gdcDl/
2016/09/29 21:54:13 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/.log gs://germline-testing/dockerflow/logs/gdcDl/
2016/09/29 21:59:13 I: Copying /var/log/google-genomics/.log to gs://germline-testing/dockerflow/logs/gdcDl/
2016/09/29 21:59:13 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/.log gs://germline-testing/dockerflow/logs/gdcDl/
2016/09/29 22:04:13 I: Copying /var/log/google-genomics/.log to gs://germline-testing/dockerflow/logs/gdcDl/
2016/09/29 22:04:13 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/.log gs://germline-testing/dockerflow/logs/gdcDl/
2016/09/29 22:04:44 I: Switching to status: delocalizing-files
2016/09/29 22:04:45 I: Docker file /gdc-data/1430729090-outputs maps to host location /mnt/gdc-data/1430729090-outputs.
2016/09/29 22:04:45 I: Copying /mnt/gdc-data/1430729090-outputs to gs://germline-testing/outputs
2016/09/29 22:04:45 I: Running command: sudo gsutil -q -m cp -L /var/log/google-genomics/out.log /mnt/gdc-data/1430729090-outputs gs://germline-testing/outputs
2016/09/29 22:04:45 E: command failed: CommandException: No URLs matched: /mnt/gdc-data/1430729090-outputs
CommandException: 1 file/object could not be transferred.
(exit status 1)
One thing that I notice in this log file is that it looks like the outputs
are expected to be available in a directory with a name following the
scheme <numerical_string>-outputs that is supposedly located in at the
root of the attached disk once the user specified command completes, but
this is not how I've specified the outputs myself so I'm not sure where
that's coming from. I've been putting all of my inputs at the root of the
attached disk and specifying individual outputs to be collected from there
as well.
As for aborting the Dataflow workflow on failure, I found this comment in
the WaitForOperation
https://github.com/googlegenomics/dockerflow/blob/master/src/main/java/com/google/cloud/genomics/dockerflow/transform/WaitForOperation.java
class:
// Check for errors and abort if that's the policy
which suggests that there might be a command line flag that could be used
to enable this functionality. Is this the case, or is there some other
mechanism used to set the policy on aborting on failure?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/googlegenomics/dockerflow/issues/9, or mute the
thread
https://github.com/notifications/unsubscribe-auth/AAiXqcxgG-Wv17PKwf1Nby6BTWLTNBV6ks5qvD43gaJpZM4KKjom
.

I found a bug in Dockerflow that was causing the pipeline to NOT abort. I've got a fix that I'll push soon.
That still won't solve whatever's causing the output file to not exist in the first place though.
First thing to check is that your Docker cmd is only referencing output files using the syntax ${OUTPUT_FILE}, where OUTPUT_FILE is the name of the outputParameter in the yaml, or the outputFile name in the Java. That's the easiest way.
If your Docker cmd script automatically generates output files using some built-in logic of its own, like adding a file suffix, or creating a file or folder in a standard place, so that you can't tell it to write the output to ${OUTPUT_FILE}, you'll need to be explicit about localPath settings.
Hope that helps!

I actually do need to be explicit about the path settings in this particular case, but it appears that the localPath settings that I've specified for the particular outputParameter are being overwritten somewhere.  I will share the workflow file, command line and logs via email shortly.

In that case, sounds like I've got another bug! I'll take a look at your example and add a test case.


Adjust DeepThunk type for changes in typescript 4.5

Description
Changes DeepThunk type to support new Restrictions on Assignability to Conditional Types in Typescript 4.5.
These changes address errors being faced updating shopify/web to 4.5.
DeepThunk was changed to separate | null | undefined from the conditional types a?b:c. It should be equivalent to the previous definition.
Type of change

[x] graphql-fixtures Patch: Bug (non-breaking change which fixes an issue)

Checklist

[x] I have added a changelog entry, prefixed by the type of change noted above (Documentation fix and Test update does not need a changelog as we do not publish new version)

Change seems reasonable. Have we tried this in web or shrink-ray either as a yalc package or a alpha?

Change seems reasonable. Have we tried this in web or shrink-ray either as a yalc package or a alpha?

Tried it locally on web via yarn link and it seemed to work for the type-checking. Haven't run the any tests, though. Should I?

Change seems reasonable. Have we tried this in web or shrink-ray either as a yalc package or a alpha?

Tried it locally on web via yarn link and it seemed to work for the type-checking. Haven't run the any tests, though. Should I?


[feat] Support for Pwsh FunctionApp deployment

Scenario
Azure Functions support Powershell Core Functions, I've written one in an azd compatible template here: https://github.com/Gordonby/functionapp-pwsh-azcli
Actual

Expected
I'd like my Pwsh function app code to be deployed to the function app.
I can see that it's not currently supported from the schema
                    "language": {
                        "type": "string",
                        "title": "Service implementation language",
                        "enum": [
                            "dotnet",
                            "csharp",
                            "fsharp",
                            "py",
                            "python",
                            "js",
                            "ts",
                            "java"
                        ]
                    },

@Gordonby, FYI, I can't seem to access https://github.com/Gordonby/functionapp-pwsh-azcli.  Is it private?

@Gordonby, FYI, I can't seem to access https://github.com/Gordonby/functionapp-pwsh-azcli. Is it private?

Oops..  It was.
I've made it public.

Hello, I would also like to deploy my code to pwsh Functions using azd.
Has anyone had the time to contribute to this feature over the last few months ?

Hello, I faced the same problems and actually could not believe that this feature was not available so I did some trial and error, with success.
According to the documentation, the following languages are supported:
"language": {
    "type": "string",
    "title": "Service implementation language",
    "enum": [
        "dotnet",
        "csharp",
        "fsharp",
        "py",
        "python",
        "js",
        "ts",
        "java"
    ]
},

And the fact that there is a py and python but no js and javascript was suspicious.
So I tried to deploy my function with py and it worked. Maybe py is a typo and should be ps?
Nevertheless py seems to stand for powershell and made it possible for me to deploy a powershell azure function via azd.


Order form bug fix

Currently throwing when wallet attempts to reconnect:
https://github.com/user-attachments/assets/8590e3b0-3a7a-465c-be10-7e35cedf6492

Given the disturbance to dev for folks, going to force merge this