Proposal: Drop smudge in favor of post-checkout hook

The improvements of the batch API and the updated fetch command are wasted on the Git smudge filters. Each checked out file has to invoke smudge, which only operates on a single file at a time. There's no parallelism here (unless Git adds it).
The post-checkout hook runs after a git checkout is run after updating the work tree. The hook can then call git lfs pull, which downloads the LFS objects for the current working directory and replaces the pointers. This means it can make full use of the batch API and concurrent transfers.
Some downsides:

Cloning is more difficult. I don't know of a way to clone with a post-checkout hook. We could solve this with a git lfs clone wrapper.
What happens if users have their own post-checkout scripts? I know that the Unreal Engine uses one, for example. Git needs a way for multiple tools to add their hooks in harmony.

Are post-checkout hooks invoked with actions like reset?

@whoisj I think so, but that's definitely a use case we'd need to check.

I've started an implementation which I'm tracking over at ttaylorr/git-lfs#1 :smile:.

Unreal Engine uses post-checkot hook for very similar usecase: they run GitDependencies.exe tool (written by Epic Games) for downloading large binaries files.
I asked them about their git-lfs plans: https://answers.unrealengine.com/questions/293214/gitdependenciesexe-vs-git-lfs.html

Finally got a chance to sit down with this, and there are a few issues I'd like to address.  The idea was to use the post-checkout hook as a means to trigger git lfs pull, which would in effect run the smudge filter in parallel.
Once I implemented this, nearly a third of integration tests were failing. I discovered the culprit is usually cloning a bare repo, and then pulling down commits from the remote later. This edge case does not invoke the post-checkout hook.  To me, this seems like a correct implementation of post-checkout since going from no commits to some commits doesn't involve a checkout.  I am not immediately sure how to solve this problem.
@whoisj brings up another issue, resets in Git also do not trigger a post-checkout.
To me, these two cases together are significant for users of Git LFS.  I'd like to discuss whether or not a post-checkout hook is the ideal way to implement this, and see if there are any other options.

Thanks for checking this out! Let's start small:

Keep the current smudge filter as the default.
You had the idea in chat of passing a flag to git lfs init to swap lfs download modes: smudge filter vs post-checkout.
Be upfront about all the caveats that we know about so users can make an informed decision if smudge filters take too long.

My pleasure! I think taking that route would be a solid move.

I looked at hooks as an alternative to filters a while ago (before I started contributing to git-lfs), and found they were too full of holes on the client side, they were clearly envisioned mostly for server-side processing. I was hoping you might find something I missed but I'm not surprised you've come to this conclusion. I agree the smudge filters should remain, as much as it's not the ideal.
Bear in mind that people using git lfs fetch --recent will avoid some of the waiting involved in future, since there's a better chance that they will already have downloaded objects by the time they check out / pull, particularly if they regularly run 'git fetch' on its own (and some tools, like SourceTree, have an option to run fetch periodically in the background).
I'm in 2 minds about post-checkout, I think it might be harder to explain to people which conditions it covers and which it doesn't, rather than simply giving people the option to turn off automatic lfs pointer->content conversion entirely & saying they have to run git lfs checkout themselves always.

A while back I started to play around with this idea, but didn't get very far. It might be a bad idea, so if you think it is or prove it is, I'm fine with that :smile:
The smudge filters could start and feed a daemon, similar to the way the credential cache daemon works. It could go something like this:

Smudge filter starts

Start daemon if it has not been started (maybe check for a lock file in .git/lfs or something)
Feed oid/pointer data to the daemon
Output the pointer data

Daemon runs

Receive input from a smudge filter
Download file, update git index (possible batching)
Exit after n seconds of inactivity (no active transfers, nothing received from filter)

The smudge filter can have logic on whether to do this or behave normally, depending on configuration. It could perhaps also make this choice based on object sizes, letting small objects operate normally and feeding large ones to a daemon. I don't know that that would be any benefit or not.
The immediate caveat with this approach is that the user feedback is poor. If the filters don't wait for the daemon to finish the repo will be in a weird, changing state until it finishes. There also may be a problem updating the index while the clone is running. I don't know for sure if git locks the index for the entire operation, or per file.
I think we can work around that, though. The smudge filter that starts the daemon ought to be able to look at .git/HEAD (or something, that stuff should all be there by that time) and know what ref it's checking out. It should be able to scan for lfs objects to know how many it's going to check out, and start up the daemon with those expectations. The daemon could then give better feedback and know when it's done. It could even tell each smudge command which number it is, so the last smudge filter to run could wait for the daemon to exit.
I'll admit that's fairly complicated. There may still be index updating issues and this approach may not work at all, but I thought I'd throw it out there anyway. I think it's possible, but the cost may not be worth it.

 What files git locks while it's calling all the smudge filters is certainly one issue, but another is simply the daemon fighting with the smudge filter (one writing the pointer data, the other writing the real data) if the retrieval is fast enough. Then there's just the inevitable nastiness of using lock files & daemons on Windows (cue all sorts of crazy problems from slight delays in locks being visibly released by the OS, leading to retries and horrid edge cases).
It would be nice if we could take the idea of a background aggregation tool but make it more predictable, say by having the entire git command book-ended so that the daemon starts at the beginning, and finishes up at the end of all the smudge filters, at that point updating the working copy and the index in bulk when it knows everything else is out the way (but being able to do anything else in parallel). I can't see a way of doing that reliably without fully wrapping the git commands though. But then again, maybe that would be ok as an option for those wanting higher perf? Each wrapper command could just use goroutines for parallelism and then call out to git for everything else, all state would be contained & no need for lock files, daemons & timeouts that I'm pretty sure will bite us in the arse in random ways.

Wrapping git would make many things we do much easier. I like that we've avoided it to this point, but a higher perf clone/checkout wrapper is probably the easiest way to go. The wrapper command doesn't need to be complex, it can set an env variable which tells the smudge filters to act as a passthrough then run lfs fetch and friends after git runs. It shouldn't need any parallelism or daemons.

Yeah I think that's the pragmatic option. Although we could still probably still make a start on the required work as the smudge filters are running; either on-demand by unambiguous output messages in stderr that we see from the wrapper (and not reflecting that back to the real stderr) or even by pre-empting the process entirely and starting work that's read-only on the core git data (not not necessarily lfs data) in parallel at the beginning of the wrapper - since in a git checkout wrapper we already have all the information we need to start doing the git lfs fetch before git even starts, only the git lfs checkout has to be done at the end.

I'm with @sinbad on this one; a background worker daemon would be awesome, but super tricky to reason about. I'm sure there are a countless number of ways that a user could mess up that process, and having to recover from all of them would be very annoying.
I'm not opposed to wrapping more git commands for users that want/need higher perf out of Git LFS.  I think we should still prefer the unwrapped commands, but as long as we're explicit about that and the wrapped commands have well-documented behavior, I think we'd be cool.

Agreed on all points. FWIW: one of my original plans over a year ago included adding support through hub. This was before I knew about the pre-push hook though. I was worried about adding an extra step to all pushes.

Sorry if this has already been answered, but in the case of smudge where you're restricted to one file at a time, why not add parallelism within each individual file download by doing chunked/segmented downloading? This is how download managers work and it helps increase download speeds tremendously.

Just to clarify, I'm suggesting that during smudge, you open several connections when downloading each single S3 object and use the Range field in your request headers to grab different portions of the file, then assemble these on the client side. See this link for details.
Having to run separate commands or use wrapper commands to enable batch downloads is a bit onerous and error prone, so it would be nice if you considered an option like this that could still fit within the smudge filter paradigm.

This is a completely separate idea from the segmented downloading:
Smudge filter starts

Just before receiving stdin, it checks whether there are any objects that need fetching (i.e. has a git lfs fetch been performed?).
If objects do need fetching, fetch all of them now, and do it in parallel. git will (hopefully?) just wait for us while we finish that.
Once the git lfs fetch is done, read stdin and give git the smudged data that it has been waiting for.
The smudge filter will continue to run. Because we have already performed a 'git lfs fetch', the smudge filter can provide the binary data immediately from the .git/lfs/objects cache.

Can someone please explain why this wouldn't work? It seems pretty obvious, so I'm assuming there's some good reason it hasn't been done already:

Thanks for the suggestion. The challenge is that Git LFS is not a long running system that keeps state. Each smudge call is a separate process with no knowledge of other processes. It doesn't know if it's smudging the first or the last file. So even if a single smudge can download a single file in parallel, it's still adding overhead by calling the LFS API and spinning up new processes for each file.
That said, it may be a good solution. I'd love to see results if you want to do some experiments.

Sure, I created this repo to benchmark things (40x10MB and 40x1MB files tracked by lfs, and 3801 C and header files from the linux kernel tracked normally by git. The LFS tracked files were created by dd'ing /dev/urandom):
https://github.com/ajohnson23/git-lfs-test-repo.git
First, I test how long LFS takes when the smudge is doing the downloading one file at a time:
git clone -n https://github.com/ajohnson23/git-lfs-test-repo.git (-n means do not run checkout, so no smudge. This is just a plain old git checkout so we're not interested in how long this takes)
cd git-lfs-test-repo && time git checkout master ( - checkout runs smudge, which downloads the LFS tracked files one at a time - )

SmudgeWithDownload:
real    12m48.320s
user    0m11.524s
sys     0m4.712s
Next I delete my old clone and run:
git clone -n https://github.com/ajohnson23/git-lfs-test-repo.git
cd git-lfs-test-repo && git lfs fetch --all (Downloads all the LFS tracked files and caches them into .git/lfs/objects - this takes approx 40 seconds on my connection)
time git checkout master ("SmudgeOnlyTime" Still running the smudge filter, still forking LFS for every file, but instead of downloading it just copies from .git/lfs/objects. This is the amount of time it would take if downloads were instantaneous)

SmudgeWithoutDownload:
real    0m0.830s
user    0m0.152s
sys     0m0.448s
Assuming something about my methodology is off, for this particular repo and my connection, download time is overwhelmingly the bottleneck and the thing that needs to be paralleled. I think multi-part downloading of single files could therefore be a huge win.

How long did the git lfs fetch take? Also, comparing a git lfs fetch --all is not an equal comparison to a plain git clone. The plain git clone will only download what's being checked out. So compare it to git lfs fetch instead.

Updated my post with my git lfs fetch time.
Note I'm not comparing git clone times at all whatsoever. If you check my commands, I only time the "git checkout master" commands.

The git lfs fetch time is not a part of my timings either -- I've only timed "git checkout master" times -- in the first case without having done "lfs fetch", in the latter case, after already having done "lfs fetch"

@ajohnson23

I think multi-part downloading of single files during smudge could therefore be a huge win in many cases.

I think you came to the wrong conclusions.
Smugle filters is stateless and make for each file (one file - one filter process):

ssh request (for ssh-downloaded repository, but not in you case) - run git-lfs-authenticate;
first https request for github api server (get real file location);
second htttps request for downloading file from CDN.

These requests are executed sequentially. The HTTPS/SSH handshake is a very expensive operation and depends on the network round-trip time.
Smugle filter is unusable slow even on the local network (I write about same problem in #376).
It would be nice if Git could:

use one process for multiple files (looks like CGI vs FastCGI);
run multiple git-lfs filters in parallel mode.

But it requires API changes inside the Git.

I understand the single file / single process nature of smudge, but I wasn't aware that LFS has a such a long sequential handshake sequence for every file, which will indeed hurt badly.
Can someone give some pointers to documents/code explaining why "first https request for github api server (get real file location);" that bozaro mentioned is necessary? Why can't a stable base path be agreed upon and used for all requests with the object URL based on the file hash which the git client will already have knowledge of? Does S3 not allow for object naming or something?

Can someone give some pointers to documents/code explaining

See: https://github.com/github/git-lfs/tree/master/docs/api

To understand the scale of the problem with handshakes I clone my micro repostory (https://github.com/bozaro/test) with single lfs file:
time git lfs fetch
Fetching master
(1 of 1 files) 33.64 KB / 40.27 KB

real	0m1.779s
user	0m0.032s
sys	0m0.008s

Yep, it's a huge problem. I know windows is super bad too, because even forking can be expensive there. Imo, the git api has needed work to better handle this problem for a long time. If that can happen, it's ideal.
If handshaking is the big bottleneck: Going back to the above dialog between sinbad and rubyist, a daemon launched at the first smudge that identifies what ref it's checking out, builds a list of the lfs objects, then does the work of gathering all of the direct object URLs in parallel so that the ongoing smudge processes can immediately have access to those rather than needing to do the sequential handshaking every time might be nice.
Taking that a step further, the daemon could also maintain an https connection and act as a local proxy for the smudge processes so that those smudge processes don't have to constantly set up a new https connection. The smudge processes could still be responsible for writing out the smudged large files so that you don't have to deal with any weird locking issues with the daemon fighting the smudge processes for ability to write/read certain files.

Can someone give some pointers to documents/code explaining why "first https request for github api server (get real file location);" that bozaro mentioned is necessary? Why can't a stable base URL be agreed upon and used for all requests with the object part of the URL based on the file hash which the git client will already have knowledge of?

Basically, the LFS API is the stable base URL. This way we don't have to tie into any specific back end storage, or dictate how the back end storage might work.
There's another logging mode people looking at this kind of thing might be interested in. You can set the env var GIT_LOG_STATS=1 for a command and it'll dump some specifics about HTTP operations to logs under .git/lfs/objects/logs/http. They'll look something like this for a download:
concurrent=3 batch=true time=1444127717 version=1.0.0
key=lfs.api.download reqheader=264 reqbody=0 resheader=121 resbody=337 restime=678553523 status=200 url=http://localhost:8080/objects/b1129123471803dc4f81c9c5a6c410061c5b1c33c736cf177619160e99f96a39
key=lfs.data.download reqheader=244 reqbody=0 resheader=117 resbody=33 restime=2646905 status=200 url=http://localhost:8080/objects/b1129123471803dc4f81c9c5a6c410061c5b1c33c736cf177619160e99f96a39

Or this for an upload (5 files, batched here):
concurrent=3 batch=true time=1444127871 version=1.0.0
key=lfs.api.batch reqheader=265 reqbody=459 resheader=122 resbody=1689 restime=670966601 status=200 url=http://localhost:8080/objects/batch
key=lfs.data.upload reqheader=284 reqbody=33 resheader=17 resbody=0 restime=2199957 status=200 url=http://localhost:8080/objects/cec8b8184af7b53f58e94b86f75162a29316f9f948a284cd2052eb6441dcc4a0
key=lfs.data.upload reqheader=284 reqbody=33 resheader=17 resbody=0 restime=2761676 status=200 url=http://localhost:8080/objects/bcc6fd9f4734af7917a1779990d20141c19f0bccef20ecfa383fdea76463e792
key=lfs.data.upload reqheader=284 reqbody=33 resheader=17 resbody=0 restime=3975767 status=200 url=http://localhost:8080/objects/0ec730dbfdf697246668c09b9bf74432ca68a44b43080dad4d113fd01649d2ed
key=lfs.data.upload reqheader=284 reqbody=33 resheader=17 resbody=0 restime=1834502 status=200 url=http://localhost:8080/objects/abc4d932160acbac152940b8893cfe41cd772a0709be203a9cb1b93cdd7a52b4
key=lfs.data.upload reqheader=284 reqbody=33 resheader=17 resbody=0 restime=1934129 status=200 url=http://localhost:8080/objects/360dff6f03c6ee9b94e4d139ae0294fd6f3cf10d727b5518296615f9817a57d7

This gives you specific request/response header and body sizes and times for api and storage operations. The caveat is that it's process based so a smudge-based clone will leave 1 log per file, so you'd want to aggregate those for analysis.

I think the implicit daemon will create a lot of problems (for example, it will block the removal of directories in Windows).
Loss of explicit complete the process of obtaining data from the server as unacceptable (mainly to build farms and build scripts). Because of this, the demon only reduces the severity of the handshake problem. But can not solve the problem of sequential queries.
So it is not clear how the demon had to deal with a filter:

TCP - is not safe in a multi-user environment;
File sockets - is platform specific.

I think it is necessary to solve the problem by updating Git API. The rest of the options seem too complicated to operate.

Yeah, ideally such things would be more of a stopgap until git apis are changed.
One issue is that this problem is far worse on windows, and the core git devs simply do not historically care about windows. I therefore do not have high hopes for the git api being changed anytime soon to accommodate this. This is why I'm proposing more drastic hacks/workarounds for this problem than I would normally entertain.
Oh well, I will just skip smudge on clone + git lfs fetch for now and hope the apis get changed one day.

:+1:

+1

+1 plz fix

Does anyone have a working post-hook solution to this problem as a stop gap until a more robust solution is implemented?

@my-digital-decay Yes, you can define the env var GIT_LFS_SKIP_SMUDGE=1 then just use git lfs pull (or use fetch and checkout separately if you'd prefer) instead to download. You can also make this setting permanent for this repo, run git lfs install --skip-smudge.

Thanks! I'll give that a try.

@sinbad, can you walk me through how to do this on a new clone? I tried:
git clone -n https://myrepo.git
That seemed to clone without engaging the lfs downloads. Then I tried:
git lfs install --skip-smudge
git lfs pull
Git lfs then downloaded the files as one batch, but then error'd out. I'm guessing this is because the HEAD wasn't checked out by git clone -n...

@WestonThayer Assuming you're using the latest git-lfs, pnce you've done the git lfs install --skip-smudge, you can just git clone https://example.git and run the git lfs pull right after. no need for using -n with clone.

@ajohnson23 thank you, I think I have my head wrapped around it now. That looked like it downloaded all my lfs files sequentially. Is there a way to parallelize at least part of it?

Add the following to your .gitconfig, and tweak concurrenttransfers to your liking:
[lfs]
×batch = true
×concurrenttransfers = 6

The default is concurrenttransfers = 3 so it should have been parallel so long as the smudge filter wasn't enabled.
Personally I prefer to leave the smudge filter enabled globally so that switching branch automatically updates my working copy, and only disable it at clone time to speed things up. To do that you just need to set the environment variable GIT_LFS_SKIP_SMUDGE=1 to have the same effect as git lfs install --skip-smudge, but just for that command. I don't like to have to remember to git lfs pull every time I switch branch.

Thanks I get it now. I posted a summary for others.


Better space distribution

Fix #23.
I did tweak some offset constants to make it look better (at least in vertical-mode) when there are not enough space. Here's a before-after screenshot:

Ideally, it should also have an option to automatically adjust all labels' font to fit. But being able to manually set font depending on your data is better than nothing. :)

You can test it out in a fixed example project in this branch https://github.com/rinatkhanov/BDKCollectionIndexView/tree/space_example
(Didn't want to mix two unrelated PRs)

Thanks for all your help with these, @rinatkhanov! I'll mark a new release today.


Installation via composer

I think in any case you should not use "dev-master", and specify stable versions.

Too lazy to update the version each time (once a year....) it is necessary, I'll improve it

You should be able to do a composer update in order to get the latest version with your pull request

Yes, with the updated composer.json package succesfull installed.


Ability to query user list for attendance

We have ninja attending to different dojos in the area. When we get a request for a white belt from a ninja we should be able to check if the ninja attended at least 5 dojo sessions, summing the attendance to the dojo in the area.
If we can't get the aggregated data from all the dojo a user is registered to, it would be enough to get the number of the events attended within a single dojo, we can then correlate the data between champions.
This would allow to have also a "score" table listing top attendant to dojos...

Apologies for the late response : I've had it open for a draft response and closed it at some point, leaving me without notifications.
Grouping of Dojo is being thought of due to regional bodies (RB) but I guess we could consider a smaller representation than RB (cc @conoramurphy). That's not in the pipeline yet, though; and it is the first time we hear of a contest-like you're designing it :+1:
In your scenario, each champion should be able to get the attendances of any of their users (that's what you see in the "Manage applications" page). To get all attendances of an event should be possible, but It's very unlikely to be able to get all attendances of all events of a Dojo in one request. You'll very likely need to loop over the Dojo's events to get that.
The reason behind is mostly the design of the API: it's REST-ish, meaning you can mostly request only one "table" at a time (exceptions to sessions because they don't exist without the event). The other reason is permissions. Only dojo-admin/ticketing admin have access to attendances, and there is no way at the moment to share those with someone else.
If you want me to help you create a small bash-script to do that, I'd be happy to :]


Implement updated version of Project::load_data in light of Library model

Context

The new addition of the Library model allows us to dramatically consolidate the processes of parsing metadata and computing files inside of  Project::load_data. This new ability to save (and later query) all relevant metadata from the input metadata files via the Sample and Library model (including the locations of data file paths), enables us to move towards a more condensed and programmatic implementation of this behavior.
Among areas that will be greatly affected includes:

Consolidating metadata parsing to focus around saving Sample and Library objects, instead of creating various ad hoc data structures in memory which are currently passed through the different stages of the metadata parsing pipeline.
Simplifying the process of combining the necessary Sample and Library metadata used in the output metadata file.
Removing all dependencies on file mapping data structures held in memory and passed from Sample file computation to Project file computation.
Removing the need to maintain workflow versions in memory between Sample and Project file computation.
Releasing the order dependency that Project file computation must come after Sample file computation.
Simplifying the way we compute files in a more dynamic and concise way via DOWNLOAD_OPTIONS, which we can more easily apply validation to later on.

These areas and others will benefit from this Epic.

PRs associated with this EPIC will be targeting  /feature/remove-file-mappings - They will be merged into dev as a single PR.

Feature branch merged in in PR https://github.com/AlexsLemonade/scpca-portal/pull/801


Fix | Misleading error message in CallableStatement

Fixed an issue where CallableStatement was not setting parameter indexes properly when columns aren't found via column name lookup. This results in the driver returning a misleading error message early instead of executing the statement and receiving the proper message from server. Addresses issue #1059.

Codecov Report

Merging #1064 into dev will decrease coverage by 0.11%.
The diff coverage is 33.33%.

@@             Coverage Diff              @@
##                dev    #1064      +/-   ##
============================================
- Coverage     53.08%   52.96%   -0.12%
+ Complexity     3159     3156       -3
============================================
  Files           119      119
  Lines         28022    28026       +4
  Branches       4688     4680       -8
============================================
- Hits          14875    14844      -31
- Misses        10912    10953      +41
+ Partials       2235     2229       -6

Impacted Files
Coverage Δ
Complexity Δ

...oft/sqlserver/jdbc/SQLServerCallableStatement.java
29.6% <33.33%> (+0.08%)
100 <0> (ø)
:arrow_down:

...om/microsoft/sqlserver/jdbc/ReaderInputStream.java
40.65% <0%> (-6.6%)
13% <0%> (-3%)

...rc/main/java/com/microsoft/sqlserver/jdbc/DDC.java
47.3% <0%> (-1.08%)
112% <0%> (ø)

...in/java/com/microsoft/sqlserver/jdbc/IOBuffer.java
60.7% <0%> (-0.8%)
0% <0%> (ø)

...ncurrentlinkedhashmap/ConcurrentLinkedHashMap.java
38.36% <0%> (-0.65%)
42% <0%> (-1%)

...c/main/java/com/microsoft/sqlserver/jdbc/Util.java
59.69% <0%> (-0.44%)
90% <0%> (-1%)

...n/java/com/microsoft/sqlserver/jdbc/Parameter.java
67.2% <0%> (-0.41%)
64% <0%> (ø)

...om/microsoft/sqlserver/jdbc/SQLServerBulkCopy.java
51.44% <0%> (-0.19%)
254% <0%> (-2%)

...rc/main/java/com/microsoft/sqlserver/jdbc/dtv.java
68.6% <0%> (-0.18%)
0% <0%> (ø)

.../microsoft/sqlserver/jdbc/SQLServerConnection.java
49.55% <0%> (-0.07%)
393% <0%> (-2%)

... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b91c7ea...1508128. Read the comment docs.


Use new way of getting build logs

Once this PR https://github.com/openshift/origin/pull/1383 gets merged and released, we need to change the way logs are retrieved.

I think you can already use the proxy method. My pull request makes that the default for osc build-logs and will make proxy keep working once the node is secured

We should use this method: https://github.com/openshift/origin/issues/1813#issuecomment-94492848


SAK-30972 Alignment issue with pagination buttons when viewing resour…

…ces from other sites

This PR needs some changes, please don't merge


De-excluding test which consistently passes now

java/lang/Class/forName/arrayClass/ExceedMaxDim.java

Conflicts to resolve


[minor] options.timeout, and request timeouts (default 60000ms)

do not merge until #97 is merged
What this times out:

[x] Time to socket open
[x] Time to first byte

What this does not time out:

[ ] Time to last byte (sindresorhus/got#161)

Ref: #96

Coverage remained the same at 100.0% when pulling 513450a97d28b50960861fffa09f256165b11725 on xbpf:feat-timeout into 1e926c524215fc8e0e21e5057946adafe4457d87 on microapps:master.

Regarding the time to last byte timeout, do you want me to implement a timeout feature for that? It will be something along the lines of Promise.race(request, setTimeout(throwError, timeout)).

Ack, I can't git.

Coverage remained the same at 100.0% when pulling 58ce2ebcf049422b2883a48e1b76d7121c7f55bf on xbpf:feat-timeout into 9c9d18e05305d51b914e5075d5ef6ebfb5bcae84 on microapps:master.

OH! There we go. I almost have 0 experience with rebasing.

Coverage remained the same at 100.0% when pulling 3f4593dae2d01e9bad1603b27949368b10699012 on xbpf:feat-timeout into 9c9d18e05305d51b914e5075d5ef6ebfb5bcae84 on microapps:master.

Regarding the time to last byte timeout, do you want me to make the timeout handle that?

No, that's a different issue.

Coverage remained the same at 100.0% when pulling f6e1f52e618046d15e7962a3e3449bda5f8606c9 on xbpf:feat-timeout into 9c9d18e05305d51b914e5075d5ef6ebfb5bcae84 on microapps:master.

Coverage remained the same at 100.0% when pulling 5dcabf13d948ad4f0926f5f9d40c005f9b957e87 on xbpf:feat-timeout into 9c9d18e05305d51b914e5075d5ef6ebfb5bcae84 on microapps:master.

Coverage remained the same at 100.0% when pulling 288796ce83fe555da58db020fcfa0808c521a692 on xbpf:feat-timeout into 9c9d18e05305d51b914e5075d5ef6ebfb5bcae84 on microapps:master.

Thank you. I'll fix few nits and release 2.0.0 soon.

Okay, thank you!


public convenience init

Could you add public convenience init so that at the same time you could set WSStream and writeQueueQOS: QualityOfService.
Thank you,

I'm gonna close this out since the new 4.0.0 version remove the use of OperationQueue from the library (and it has been a long time). Any follow up question regrading that please open a new issue with the 4.0.0 changes. Thanks!


[Proposed Changes] - Empty Section?

Link:
https://www.generativedesign.org/03-hello-gd-for-revit/03-05_refinery-toolkit/using-the-refinery-toolkit
Tasks:

[ ]  Section is empty?

This should be fixed on the next build of the gitbook. Need to check


Fix path in vcproj for SVM

Cherry-pick from https://github.com/oneapi-src/oneDAL/pull/670

/intelci: run


[stable/prometheus] Add configFromSecret support to StatefulSet alertmanagers

Is this a new chart
No
What this PR does / why we need it:
Updates the StatefulSet template for alertmanager to use the alertmanager.configFromSecret setting if it is present. This behavior is already supported by the Deployment template.
Which issue this PR fixes

fixes #16913

Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] DCO signed
[x] Chart Version bumped
[x] Title of the PR starts with chart name (e.g. [stable/mychartname])

/assign

/ok-to-test

/lgtm


Stack exaustion while value_from(std::filesystem::path)

PLEASE DON'T FORGET TO "STAR" THIS REPOSITORY :)
Are you inviting me to have multiple account or more friend ?
Version of Boost
1.84
Steps necessary to reproduce the problem
#include <boost/json/src.hpp>
#include <filesystem>
#include <iostream>

namespace bj = boost::json;

int main() {
	std::filesystem::path p = "/cry";

	auto ref = bj::value_from(p);
	auto ser = bj::serialize(ref);
	std::cout << ser;
}

if the path is EMPTY it works fine, but if there is something it leads to a stack exausting crash, remaining struct inside those 2
loop (first one is gcc the other clang, quite similar but a bit different)

Also using
void tag_invoke(const bj::value_from_tag&, bj::value& jv, const std::filesystem::path& t) {
	jv = t.string();
}

std::filesystem::path tag_invoke(const bj::value_to_tag<std::filesystem::path>&, const boost::json::value& jv) {
	std::filesystem::path path = std::string_view(jv.as_string());
	return path;
}

or

void tag_invoke(const bj::value_from_tag&, bj::value& jv, const std::filesystem::__cxx11::path& t) {
	jv = t.string();
}

std::filesystem::path tag_invoke(const bj::value_to_tag<std::filesystem::__cxx11::path>&, const boost::json::value& jv) {
	std::filesystem::path path = std::string_view(jv.as_string());
	return path;
}

Does not help (in fact looks like they are not even used...)

The solution is to put the tag_invoke INSIDE the boost::json namespace...
This already happened in another case, for std::chrono,
I think it should be worth to add a note into this page
https://www.boost.org/doc/libs/1_84_0/libs/json/doc/html/json/conversion/custom_conversions.html
To express that, sometimes... you have to put in namespace to help it!

The solution (to the error) is to not treat path as a sequence. It's not what users want anyway (everyone would prefer conversion to a string). This will result in value_to<path> being a compilation error.
The next step will be to add a conversion category for paths.

The solution is to put the tag_invoke INSIDE the boost::json namespace...
I think it should be worth to add a note into this page

From that page:

embed the user-defined type into the arguments list (e.g. by using a tag type template such as value_to_tag) so that its associated namespaces and entities are examined when name lookup is performed.

Yes maybe just drop a few lines about that specific example ?
Is ok if I do a pull request for https://github.com/boostorg/json/blob/db92f8c22360990f450fe27b86ea1a5830b5cf05/doc/qbk/conversion/custom.qbk#L54 ?

You can create a PR, but before please look for a good place to put the note.


Use unique_hash to prevent duplicate ProjectUser counts

This morning we had about 14,000 duplicate of each of (28k total):
ProjectUser.delay(:priority => INTEGRITY_PRIORITY).update_observations_counter_cache_from_project_and_user(po.project_id, obs.user_id)
ProjectUser.delay(:priority => INTEGRITY_PRIORITY).update_taxa_counter_cache_from_project_and_user(po.project_id, obs.user_id)

Not sure exactly where it originated, but this was the project http://www.inaturalist.org/projects/whio-forever. This is a good place for the unique_hash to prevent duplication

I was getting the heavy traffic error this morning, perhaps this is why
On Thu, Oct 22, 2015 at 10:35 AM, Patrick Leary<EMAIL_ADDRESS>wrote:

This morning we had about 14,000 duplicate of each of (28k total):
ProjectUser.delay(:priority => INTEGRITY_PRIORITY).update_observations_counter_cache_from_project_and_user(po.project_id, obs.user_id)
ProjectUser.delay(:priority => INTEGRITY_PRIORITY).update_taxa_counter_cache_from_project_and_user(po.project_id, obs.user_id)
Not sure exactly where it originated, but this was the project
http://www.inaturalist.org/projects/whio-forever. This is a good place
for the unique_hash to prevent duplication
—
Reply to this email directly or view it on GitHub
https://github.com/inaturalist/inaturalist/issues/755.

--
Charlie Hohn
Montpelier, Vermont


Need better diagnostics

I spent a large part of yesterday trying to get html-webpack-plugin working with hot loading. A SO Question convinced me it should be possible and in the end it turned out to be very sensitive to the index.html path. I had to handle it differently between hot serving and build. At least I did get it working
Any sort of error message would really help with this sort of problem. Form SO it's clear others have problems.
Here is my config
https://github.com/OpenDirective/brian/blob/master/package.json https://github.com/OpenDirective/brian/blob/master/client/webpack.config.js
In particular build:
    plugins: [new HtmlWebpackPlugin({filename: path.join(PATHS.dist, 'index.html'),
                          template: path.join(PATHS.src, 'index.html'),
                          inject: true
                          }),
              new CopyWebpackPlugin([{from: path.join(PATHS.src, 'favicon.ico'), to: '..'},
                          {from: path.join(PATHS.src, 'CNAME'), to: '..'},
                                {from: path.join(PATHS.src, 'img'), to: '../img'}]), // abs paths to: don't work
              new webpack.NoErrorsPlugin(),
              new ExtractTextPlugin('../css/[name].css'),
              new webpack.optimize.UglifyJsPlugin({minimize: true})
  ]

and dev serve:
    plugins: [
      new HtmlWebpackPlugin({filename: 'index.html',
                              template: path.join(PATHS.src, 'index.html'),
                              inject: true
                            })
    ]

Thanks

Cool thanks I don't understand the problem but please open a pull request for a better error message - would love to merge it


Add missing return types

Just a small pr to add missing return types in commands.
Because in some commands it was present in other not.
I'll also check for final class in another pr.

Oups i see some wrong commits names and some merge commits, i'll rebase this to clear :smile:


Failure to launch pods in EKS (containerd)

Looks like the launcher fails to run Docker due to missing privileges. Oddly enough the capabilities as reported by containerd look fine. $UID says 0.
Originally tried a few clabernetes versions (as I was suspecting some capabilities issue) and legacy iptables, no luck. I'm attaching a debug session showing where it fails and the capabilities applied to the containers. Any pointers appreciated.
docker.txt
node.txt

👋
hey there! can you confirm that the deployment is privileged? if it is (which should be the default if you are on latest/greatest nvm I can read. also I dont remember why 0.0.30 is a pre release, but guess that shouldn't matter!) then I suspect I have nothing useful to add!
I dont have any eks access so not sure what else would be funky there but assuming you have a privileged pod you "should" be able to do whatever of course.
generally I'd try to kill the controller (so it lets you mess about without re-reconciling -- also there is a label you can add to topology to tell controller to not reconcile: clabernetes/ignoreReconcile) and then change entry point of launcher to sleep infinity and exec on and play around and see if anything else interesting pops up

Hey, thanks! Got the privilege issue sorted (basically --profile=netadmin), but it still fails exporting the image. I tried changing the discard_unpacked_layers setting on the node but nothing changed. Looks like it finds the image after the puller downloads it but cannot move it over. Is there any way to mount the pull secret on the launcher?
`    INFO |               clabernetes | image "IMAGE" is present, begin copy to docker daemon...
INFO |               clabernetes | time="2024-05-01T14:32:12Z" level=fatal msg="failed to resolve reference "IMAGE": unexpected status from HEAD request to https://REGISTRY: 401 Unauthorized"
WARN |               clabernetes | image re-pull failed, this can happen when containerd sets `discard_unpacked_layers` sets to true and we don't have appropriate pull secrets for pulling the image. will continue attempting image pull through but this may fail, error: exit status 1`

Oh this is what gives us pain...
But before we get to the image pull, can you elaborate for my understanding what was the issue with the privileged and profiles?
I would like to document this
As for the image move we have been trying to chat with various people from cloud providers to have an option to allow copying the image that was pulled to a worker, but it seems unless a cloud provider lets you configure the containerd runtime, there is not much we can do...
At the same time, the image "move" albeit being super annoying shouldn't stop the image from being pulled into a launcher, the launcher will have to pull it again, instead of copying it from the worker.

--profile=netadmin

as in kubectl flag or something else? the deployment the manager spawns should be privileged by default so there "shouldn't" be any other things you need to do unless there is some aws/eks specific thing you had to do (and if so please elaborate more for future me/folks :))

Is there any way to mount the pull secret on the launcher

not exactly, no. you can mount a docker config which could include whatever auth you want. otherwise this setup is intentional so that we never have pull secrets on the launcher and so that we can use the host nodes to cache the images. this whole layer nonsense really blows the spot though. but... like roman said this shouldn't stop things though (assuming image is publicly pullable), it will just stop us from using the host nodes as the cache/letting k8s pull stuff work, the launcher will then just fail back to pulling the image directly (like from its docker daemon I mean).
if you re looking for the docker daemon config you set it like spec.imagePull.dockerDaemonConfig and it points to some secret in the namespace of your topology then we just mount it for you.

quickly replying here:
Privileged profile: I had been fiddling with pod security standards as per https://kubernetes.io/docs/concepts/security/pod-security-standards/. I ended up applying the privileged profile to the clab namespace. Also when debugging with kubectl debug it needs passing the profile, e.g., kubectl debug -n c9s-ceos --profile= [ sysadmin | netadmin ].
Pull secrets: The registry is private (has to be in this case). The puller works fine with the pull secrets provided, it's the moving over to Docker from the launcher that fails. I found spec.imagePull.dockerDaemonConfig by looking at the code, the issue with passing Docker daemon config is that there's nowhere to provide auth creds in there (that I know of).

re priv -- ok, so you had to set pss stuff which probably had some tighter stuff by default for eks I assume; but after this, the launcher/docker should run fine right? (assuming this since the pod should be privileged)
yeah the discard_unpacked_layers thing is the bane of our existence. you mentioned you tried to set it but it still fails? it "shouldn't" if that is set nicely. do you manage the nodes? did containerd get restarted and the image re-pulled? I imagine if the image was present even if it was all set nicely the image would have to get re-pulled so it has all the layers.
regarding secrets/daemon config: blah, yeah probably this is my bad since I was thinking you could stuff auth stuff in daemon config but probably never tested since I just use pull through thing anyway. I can add spec.imagePull.dockerConfig and have it be basically same setup as the daemon config -- then you can set auth stuff that way. in the meantime could you try to pop on the launcher and do a docker login and pull your image just to make sure all that part works?
thanks!

Privileged profile: yes, that's correct. All working fine now on that front.

nice, copy, thanks!

discard_unpacked_layers

continues to be the bane of our existence, and now yours too maybe 🤣

spec.imagePull.dockerConfig

cool, will try to work that up over the next week or so. not hard, just need some time to do it! if you are up for a pr that would be cool too -- basically would just follow same pattern as the daemon config, so shouldn't be too terrible.
im not 100% im tracking on the remaining, but if/when the launcher fails that container should restart so I dont think there should be any cruft left. if you are talking about the services (vx- blah) then those should be reconciled every time the topology is reconciled (and deleted via owner ref when topo is deleted) so those shouldn't have cruft either.
I guess the general thing here is: the launcher should crash if it cant continue, and that crash should let us have a new container w/ fresh stuff. if thats not the case we can fix that but I think/hope that should work. by deleting the pod you are doing the same thing that the launcher would do when it crashes so I think/hope that will be ok. maybe the pod didn't restart because you changed entry point or something while troubleshooting?
but yeah if you can document and get another issue rolling that would be cool! then we can keep this open to track the docker config bits! thanks @flaminidavid !

I'm running into some more EKS fun given the particularities of my setup. I'll report back when those are sorted (and open another issue for the interface cleanup on failure, it's still happening, although I believe the container failures are caused by something else). Thank you!

@flaminidavid fyi the docker config stuff is in v0.1.0 now -- so that should be one less problem for ya at least 🙃

Thanks a million. I got my setup to work consistently in EKS, still using a local registry for now. The last tranche of launcher failures was caused by a background process that identified some router container processes as a threat (can't talk specifics here). Just in case anyone else runs into this and rabbit holes debugging containers booting, container processes, and VXLAN setup: it wasn't any of that. The cleanup probably works well in any other situation. I think we can close this one.

Good to know @flaminidavid
since you had a good run with EKS do you think we can go as far as say that c9s runs on EKS without manual tinkering considering it is a clean EKS deployment and priv. pods can run there?

I'd say so, yes.
Aside from the discard_unpacked_layers issue that we should be able to work around by passing Docker config everything else seems to work fine in general. "Just make sure nothing else is messing with the cluster" would be my advice 🙃.

sweet, ty.
I will have to try onboard a simple Nokia SR Linux topology there to make sure I capture the docs steps for others to have a smoother, documented experience.
I will close this now.

Just confirming: overriding Docker config via spec.imagePull.dockerConfig worked just fine (and switched back pullThroughOverride from never to auto). This is working with AWS ECR, too.


False positive for curl command in sh file

Describe the bug
RESPONSE=$(curl -sSL -H "${AUTH_HEADER}" -H "${HEADER}" --user "${GITHUB_ACTOR}" -X GET --data "${DATA}" ${PULLS_URL})
case-police tells me that sSL should be SSL
though this is a curl shorthand rather than a word
Reproduction
https://stackblitz.com/edit/vitejs-vite-5bjczq?file=run.sh
System Info
System:
    OS: Linux 5.15 Ubuntu 20.04.4 LTS (Focal Fossa)
    CPU: (16) x64 AMD EPYC 7B13
    Memory: 19.95 GB / 62.81 GB
    Container: Yes
    Shell: 5.0.17 - /bin/bash
  Binaries:
    Node: 16.16.0 - ~/.nvm/versions/node/v16.16.0/bin/node
    Yarn: 1.22.17 - ~/.nvm/versions/node/v16.16.0/bin/yarn
    npm: 8.11.0 - ~/.nvm/versions/node/v16.16.0/bin/npm

Used Package Manager
yarn
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guide.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A, please open a GitHub Discussion instead.
[X] The provided reproduction is a minimal reproducible of the bug.

I suppose that's one way to fix it :p


Update street column in us/sc/jasper.json

The street labels currently use the FULL_STREET field but this field contains everything including the street number. Change to get only the street label using other fields so the output is consistent with other sources.

No idea why this failed, I kicked it off again.

@ingalls any idea what the error AssertionError: R2_BUCKET must be set means?

@anthony-moreau-qapa can you create a branch for this repo w/o forking?  Otherwise I can do it.


Provide the instructions to install SQLCMD on FROM mcr.microsoft.com/dotnet/aspnet:6.0

Especially for troubleshooting purposes, mssqltools are vital to install to see if a container can connect to the SQL properly.
Unfortunately I couldn't install it on Debian 11, and I appreciate if you may provide necessary steps to install it.

Hi @cilerler! If you deploy and run the mssql-server container by default it also installs the mssql tools. We dont have a package available on mcr only to install tools, but you can build a custom image which only has mssql tools and a sample dockerfile is available here for reference https://github.com/microsoft/mssql-docker/blob/master/linux/mssql-tools/Dockerfile

That is not working, that is the reason, I'm asking for updated
documentation, please.
On Sat, Jan 22, 2022, 9:23 AM amvin87 @.***> wrote:

Hi @cilerler https://github.com/cilerler! If you deploy and run the
mssql-server container by default it also installs the mssql tools. We dont
have a package available on mcr only to install tools, but you can build a
custom image which only has mssql tools and a sample dockerfile is
available here for reference
https://github.com/microsoft/mssql-docker/blob/master/linux/mssql-tools/Dockerfile
—
Reply to this email directly, view it on GitHub
https://github.com/microsoft/mssql-docker/issues/736#issuecomment-1019279985,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAC6HJQTN563ZNWRKKBEXZDUXK4WTANCNFSM5MPJRQUA
.
You are receiving this because you were mentioned.Message ID:
@.***>

@amvin87, please do not close it.
I'm already aware of it that mssql-server comes with it, and already tested to do it the way that Dockerfile does, and it didn't work, and that is the reason I posted it here.
Just change the first line FROM Ubuntu:16.04 to FROM mcr.microsoft.com/dotnet/aspnet:6.0 and you will see that it won't work due to https://github.com/microsoft/mssql-docker/blob/37532c96cd71e95b5c93747669bd1fa606019ad7/linux/mssql-tools/Dockerfile#L12


This plugin doesn't work for AW-* tags

It creates an error [nuxt] error caught during app initialization Error: 'AW-*' is not a valid GTM-ID (/^GTM-[0-9A-Z]+$/). Did you mean 'GTM-*'?, which is issue one.
However, it also creates very unintuitive error when using Vuetify for a project, namely nuxt.js?v=49ff4004:96 [nuxt] error caught during app initialization Error: [Vuetify] Could not find defaults instance. Lots of functionalities don't work with that setup.
Repo to reproduce: https://github.com/Doyban/Doyban-Website/tree/cab0e05bd28d2e11241b88a72087c001dc2ba188
I tried Node 16/18/20, Vuetify with/out SSR, and the same issue.

PS. What you need apart from yarn && yarn install is an environmental variable in .env file GTM_ID=AW-*, to reproduce.

ahh, and tried with runtimeConfig.public and without as well!


Make ShieldView more customizable

Description of Change
Make Shield subject more customizable, details in issue.
Bugs Fixed

closes #862
API Changes
Please refer to linked issue.
PR Checklist

[ ] Has tests (if omitted, state reason in description)
[x] Has samples (if omitted, state reason in description)
[x] Rebased on top of main at time of PR
[x] Changes adhere to coding standard

[ ] Updated documentation

Remains whether we add the possibility to customize the fond properties separately Subject and Status or no.

Would you also be able to update the Docs? :)

@jfversluis sure it is in my mind I will asap.

thank you so much for this one.I really needed the ability to style the subject too, it now makes fully styleable


update git clone instructions

HTTPS is the preferred way to clone a repo by GitHub, as SSH only works if you have the key setup in your account.

Appears when you edit in GitHub it assigns a different email to the commit, fixing...


Documentation and configuration overhaul

Ongoing work. Don't merge yet.

Attached are test the results
test-runner-2019-02-17 12-17-25.txt


Add formatting option for whitespace

Formatting adds these three wihitespaces that I personally don't like:
"3": { "name": "punctuation.definition.markdown" },
      ^                                         ^ ws
"fileTypes": [ ],
              ^ ws

Maybe add a configuration option "whitespaceInsideBrackts": {"type": "boolean"}? I can make a PR, if you like.

Good idea
PR would be welcomed.
even if it was just the configuration side of it
I could do the format part of it if need be
at some point, I would like to overhaul the formatter
as it doesn't handle ERROR's very well
format selection is too vague
and it isn't particularly fast either atm
the main files would be DocumentFormattingEditProvider.ts, extension.ts and package.json

@RedCMD speaking about formatting. Is there code style for this project?

there's no config file atm
that format file is rather out of date compared to everything else (limited semi-colons etc)


Fixes #27625 -- Made LazySettings cache attributes in its own dict

Ticket

I'm not sure if this needs a release note. It shouldn't affect applications that use settings in the documented way, but hacks like Instagram's are now unnecessary.

Nice, thanks!

I added a release note, just in case

Did you find evidence of widespread usage on GitHub? That's the only case when we document private API changes.

I think it's impossible to do an actual code search for it, but there are lots of examples of code that interacts with the _wrapped attribute and __dict__: https://github.com/search?l=Python&q=settings._wrapped&type=Code&utf8=✓

Looking again at the results I see on code search, many are copies of Django itself, and I haven't found one that will be definitely broken by this change, plus the Instagram workaround is compatible too, so I've decided to remove the release note.


ENHANCEMENT: Add faclassExpression property to the Font-awesome component

Currently, it is not easy to change the icon that is displayed at runtime unless you use code.
Allowing a dataprovider as well as a text class will make the component much more versatile.

We refactored a bit our components and added an imageStyleClass to the
Data  Label component from the Bootstrap Components package
This will be available in the next release.
So my suggestion is to use the Data Label instead because that also supports a dataprovider.


StorageException: Reuse code from RequestFailedException

This PR modifies the creation of StorageException to reuse code of its baseclass RequestFailedException to initialize the shared fields. In particular, this fixes the issue that ErrorCode was not set.
Pull Request Checklist
Please leverage this checklist as a reminder to address commonly occurring feedback when submitting a pull request to make sure your PR can be reviewed quickly:
See the detailed list in the contributing guide.

[x] C++ Guidelines
[x] Doxygen docs
[ ] Unit tests
[x] No unwanted commits/changes
[x] Descriptive title/description

[x] PR is single purpose
[ ] Related issue listed

[x] Comments in source
[x] No typos
[ ] Update changelog
[x] Not work-in-progress
[ ] External references or docs updated
[x] Self review of PR done
[ ] Any breaking changes?

@JeffreyRichter, @Jinming-Hu - what are your thoughts here?

The latest thinking (and what we're doing in Go) is that Core defines a single RequestFailedException and this is thrown whenever a HTTP response has a non-2xx status code. Inside this, there is the raw HTTP response, a field with the HTTP status code and a string field that returns an operation-specific error code. Azure has a standard error response see here. The 2 fields I mention are the ONLY fields that are part of Azure error response contract and these are the ONLY 2 fields customers could reference in their code to make an error recovery decision. It is also useful to have the exception type's ToString() method return the same string that would be passed to the logger. We expect customers to call this method to record/log the failed exception in order to fix their code.
How close can we get to this without breaking customers? Perhaps we make Core's RequestFailedException do what I mention above and then StorageException can simply derive off of it and inherit the behavior in its entirety? (And add other stuff if necessary to avoid breaks).

I sure would be interested to bring an improvement to master but my feeling is that we need more input from the team owning the library on how the change should be laid out.


link README to Postman public collection

This PR links to the Postman public workspace collection. The collection also has documentation that is picked up from the DVGA solutions page.
Removed the collection.json we had in the repository, so we don't have to maintain the collection in two places.
Users can either import the collection by clicking on the Run in Postman button or fork it so it's easy to pull changes if the base collection ever changes.
Right now I have access to the public workspace and can edit the main collection, if you'd like me to send an invite to you so you can maintain it too, let me know.

awesome :)


Initial transcribe modules

Current Coverage via 91affb8 is 32.85%

Merging transcribe into master will change coverage by -67.15%

@@            master   transcribe   diff @@
===========================================
  Files            7           16     +9
  Stmts           56          274   +218
  Branches         0            0
===========================================
+ Hit             56           90    +34
  Partial          0            0
- Missed           0          184   +184


chore: change ecmaVersion to 2017 in .eslintrc.json

Current coverage is 81.13% (diff: 100%)

No coverage report found for master at 32064c9.

Powered by Codecov. Last update 32064c9...7d91b59


E2: changed() functionality

So I am not exactly sure about the changed function working, unless it is its intended way to work like that. Basically in code terms what is my idea how changed should work is
changed(Value)
should work the same way as
$Value&1
I am working on a code, where some values when changed triggers some timers. After some time I found the changed() function doesn't return 1 where I would expect it to do (as I stated above). I wrote a piece of code (changed_test.txt) where the results showed me that the idea I have, doesn't work.
Printed results where basically the value of changed() function result should be the same as delta result, but they aren't:
//First execution
1	=	First: changed=1/Delta=1
//interval 100
1	=	interval: changed=1/Delta=1
2	=	interval ang(2): changed=1/Delta=1
3	=	interval ang(1): changed=1/Delta=1
//interval 1000
1	=	interval: changed=0/Delta=1
2	=	interval ang(2): changed=0/Delta=1
3	=	interval ang(1): changed=0/Delta=1
I feel like that the changed() function stores an array of values that have been assigned to that variable somehow, and detects if the value isn't in the array? Is this the the intented functionality or is it a bug?

The delta operator gives the difference between the last value and the current value. It does this by storing the old value whenever it's assigned to.
The changed functions work by storing the expression they're used on and the value it returns. Every instance of the changed function uses a different expression, even if it looks the same as another one. It doesn't look at the variable used in it or anything.
Another thing is that the changed operator will always return 1 on its first run.
For example,
for(I = 1, 2)
{
    print(changed(I))
    print(changed(I))
}

The output of this will be
1
1
1
1

Even though the value doesn't change between the first and second print it still prints 1 because they're separate expressions.

Oh, I see. I thought that it works pretty the much the same as the delta functionality with extension to custom expressions which aren't a variable, but here it stores the value it returned for every changed() used separately in the code (every instance as you stated, where I kinda struggled to understand).
Basically I have a bigger code, with timer sections, where each section run only when needed. What made me ask about the changed() is that, a section A set a persist variable then stopped running and switched to section B. The section B changed the same variable to a different value, stopped running and switched back again to the section A. Section A set value the variable back to the value which was set the last time in section A before the changed() was ran, therefore the changed() return 0 where I was expecting it to return 1.
I guess I'll just have to use something else, like the delta itself although its more expensive. Thank you for explaining, very much appreciated.


Add samples for maps data-plane

Codegen will be added here: pull/22269
This will most likely wait until that enters first.

Hi @JonathanGiles, Daniel here with the Azure Maps team. These examples were generated for Track1, so they need updating to Track2 and with new generated code. I will probably open another PR with the generated code + examples all in one. I will merge this to the current development branch, and if it all goes well, I'll ping back here and this PR can be closed.


Trying to get particle to publish an event.

Hello rlisle,
I have duplicated your project here to the best of my ability and I can send a test call from the Alexa Developer Test portal to the aws lambda function and get the expected response.  I have loaded the Arduino example.ino to the particle and added a Particle.publish ("It works."); .
My hope was to see a published event in the Console Log when the alexa skill is tested through the developer portal.  However I am unable to get a published log from aws lambda calls.  I then removed the example and replaced with nothing.
void setup(){}
void loop(){}
and was still able to test from the alexa developer portal to aws lambda and get the successful response from my aws lambda function.
Why is this not talking to the particle and still acting happy?  Shouldn't I be getting and endpoint error.
this is how I modified the example.  any ideas?
void setup() {
Serial.begin(57600)
Serial.println("ParticleExample started.")
Particle.function("myFunction",myFunction);
}
void loop() {
// put your main code here, to run repeatedly:
}
int myFunction(String command) {
Serial.println("myFunction called.");
Particle.publish("itWorked");
// We don't need to return anything.
// Just return an easy to recognize number for testing
return 123;
}

Alright, figured out what I had wrong,  I had the incorrect access token.

Glad to hear it. I'll be releasing additional code that adds iOS support and an Alexa Smart Home skill shortly.


Unrepair Math-84

jGenProg arrives to repair with Gzoltar, but not with Flacoco (which does not fail)
Logs for the two executions, first Gzoltar, then Flacoco
https://github.com/martinezmatias/astor/runs/3676645094?check_suite_focus=true

This is not an issue with flacoco.
The location that is patched through GZoltar is the following:
location= org.apache.commons.math.optimization.direct.MultiDirectional
line= 90
lineSuspiciousness= 0.707

Flacoco identifies this line, with the same suspiciousness value: org.apache.commons.math.optimization.direct.MultiDirectional@-@90,0.7071067811865475
The difference is in the "priority". For GZoltar, this line is the 7th, whereas for Flacoco it is the 11th:
GZoltar:
[21396] INFO GZoltarFaultLocalization - Suspicious:  line org.apache.commons.math.optimization.direct.MultiDirectional l: 66, susp 0.7071067811865475
[21396] INFO GZoltarFaultLocalization - Suspicious:  line org.apache.commons.math.optimization.direct.MultiDirectional l: 69, susp 0.7071067811865475
[21396] INFO GZoltarFaultLocalization - Suspicious:  line org.apache.commons.math.optimization.direct.MultiDirectional l: 70, susp 0.7071067811865475
[21396] INFO GZoltarFaultLocalization - Suspicious:  line org.apache.commons.math.optimization.direct.MultiDirectional l: 73, susp 0.7071067811865475
[21396] INFO GZoltarFaultLocalization - Suspicious:  line org.apache.commons.math.optimization.direct.MultiDirectional l: 74, susp 0.7071067811865475
[21396] INFO GZoltarFaultLocalization - Suspicious:  line org.apache.commons.math.optimization.direct.MultiDirectional l: 89, susp 0.7071067811865475
[21396] INFO GZoltarFaultLocalization - Suspicious:  line org.apache.commons.math.optimization.direct.MultiDirectional l: 90, susp 0.7071067811865475

Flacoco:
org.apache.commons.math.optimization.direct.MultiDirectional@-@45,0.7071067811865475
org.apache.commons.math.optimization.direct.MultiDirectional@-@46,0.7071067811865475
org.apache.commons.math.optimization.direct.MultiDirectional@-@47,0.7071067811865475
org.apache.commons.math.optimization.direct.MultiDirectional@-@48,0.7071067811865475
org.apache.commons.math.optimization.direct.MultiDirectional@-@66,0.7071067811865475
org.apache.commons.math.optimization.direct.MultiDirectional@-@69,0.7071067811865475
org.apache.commons.math.optimization.direct.MultiDirectional@-@70,0.7071067811865475
org.apache.commons.math.optimization.direct.MultiDirectional@-@73,0.7071067811865475
org.apache.commons.math.optimization.direct.MultiDirectional@-@74,0.7071067811865475
org.apache.commons.math.optimization.direct.MultiDirectional@-@89,0.7071067811865475
org.apache.commons.math.optimization.direct.MultiDirectional@-@90,0.7071067811865475

Is that a timout issue then? (Astor does not have the time to reach the 11th line?

Is that a timout issue then? (Astor does not have the time to reach the 11th line?

In this case yes (https://github.com/martinezmatias/astor/runs/3676645094?check_suite_focus=true#step:8:2326).
A similar issue might arise with the limit of generations.


Convert button in adds fund modal bug

Convert button needs to change to processing once hit (or signing tx is hit)...currently stays as convert - issue is user can hit it multiple times. User has no idea if tx went through.

Processing button appeared after MM tx confirmed....


Promise Cancellable Testing

Description
The PromiseCancellable requires testing in cancellation behaviour and its behaviour in DAGs
Issues Fixed

Related https://github.com/MatrixAI/Polykey/issues/297

Tasks

[ ] 1. ...
[ ] 2. ...
[ ] 3. ...

Final checklist

[ ] Domain specific tests
[ ] Full tests
[ ] Updated inline-comment documentation
[ ] Lint fixed
[ ] Squash and rebased
[ ] Sanity check the final build

Still need to do DAG rejection where we want to test local and global side effects.
One of the trickiest parts is where there's an intermediate promise. In that area, early rejection can be a problem, because P3 is already rejected.
One way to avoid early rejection with P1, P2, and P3 chain. Is to pass a signal handler that doesn't do anything. This ends up causing P3 cancellation to wait on P1, and then leave it to P2 to decide what to do.
The alternative is to put a catch on P2 and handle any rejections that occur there with the understanding that P3 may already be rejected.

I've tested with normal promises and in fact this is the default behaviour of promises if rejection were to occur early:

import process from 'process';

process.on('unhandledRejection', (r, p) => {
  // @ts-ignore
  console.log('UNHANDLED', r.name, r.message);
});

class CustomPromise<T> extends Promise<T> {

  public reject;

  constructor(executor) {
    let reject_;
    super((resolve, reject) => {
      reject_ = reject;
      executor(resolve, reject);
    });
    this.reject = reject_;
  }

}

async function main () {

  const p1 = new CustomPromise<string>((resolve) => {
    setTimeout(() => {
      resolve('p1 result');
    }, 100)
  });

  const p3 = p1.then(() => {
    const p2 = new CustomPromise<string>((resolve, reject) => {
      setTimeout(() => {
        reject(new Error('P2 REJECT'));
      }, 100);
    });
    return p2;
  });

  const p5 = p3.then(() => {
    return new CustomPromise<string>((resolve) => {
      resolve('lol');
    });
  }, (r) => {
    console.log('P4 is never produced');
    throw r;
  });

  // @ts-ignore
  p3.reject(new Error('P3 REJECT'));

  try {
    const r = await p5;
    console.log(r);
  } catch (e) {
    console.log(e.name, e.message);
  }

}

void main();

It does appear that P3(P1 -> P2) means that cancelling P3 if intended to chain, should actually be cancelling P1. Not P2, because P2 doesn't exist yet.
Now if P1 is already done by the time P3 is cancelling, then the cancel signal would be ignored by P1 because double rejection is a noop, then the computation to P2 can be informed to also reject.
If we want that the default rejection not to cause problems, it seems that we would want:
p3 = p1.then(() => p2, reject?);

To be desugared to:
p1.then(() => p2).catch(reject?)

So that it is possible for p2 rejection to occur and just be caught...
Let me try this..

It actually works. Simply by sticking catch(r => throw r) after the then, it ends up handling the rejection and converting it to an exception.
In fact because the default handler is just to throw the reason. It's simple enough to just do catch().
So:
const p3 = p1.then(() => p2).catch();

Is enough to allow p3 to do an early rejection without being affected by an unhandled promise rejection in p2.
This does not work if instead we instead did:
const p3 = p1.then(() => p2.catch());

Part of the reason is that catch itself produces a new promise so that p3 is really rejecting the return of the catch, and not the return of the then. It appears by doing this, there's no more unhandled promise rejection, allowing the p2 function to also reject if it wants to.

We will need to test this for all the other binders like catch itself.
If catch itself returned a promise, that also gets aborted, we would want to prevent any kind of unhandled promise rejection there.
At the same time we may have to change to using the species feature so that super refers to the native Promise, so I can avoid too many intermediate useless AbortController objects.

I have found I don't need the catch() trick anymore. This is because by using PromiseCancellable.from, it ends up adding a handler to any rejection on the intermediate p2 promise for both then and catch and there are no more unhandled promise rejections.

Furthermore I have found the species is required but it doesn't actually do what I thought it did. It turns out super.then() now returns Promise, while super.catch() actually returns PromiseCancellable. This is pretty strange.

In fact, it doesn't change the fact that PromiseCancellable.all still returns an instance of PromiseCancellable. So I'm actually not sure what the species is really doing then.

After checking some issues for the @@species, it seems the docs isn't comprehensive on how it works. It doesn't actually affect alot of static calls.
See: https://github.com/tc39/ecma262/issues/151#issuecomment-153432512
What's interesting is that then still uses it, but catch doesn't... at least in my tests.

Ok, the reason why super.catch and super.finally appear to produce PromiseCancellable already, is because they both actually end up calling this.then.
So because this.then already produces a PromiseCancellable, the result of catch and finally also produce PromiseCancellable.
So now I'm checking if I can optimise this so it's not producing an additional PromiseCancellable for no reason.
One thing that is challenging is the fact that because then is already called, and the default behaviour is connect its signal to cancellation, it appears that we should actually replace the abortController when catch or finally is called.

So basically all I have to do is to mutate the promise returned by this.then and then proceed to use it. The tests continue to pass.

It appears that the PromiseCancellable.all does end up using PromiseCancellable.then, however it doesn't actually return the result here. Instead a new promise is constructed for this, thus the from isn't being used.
I can imagine something like:
all(p) {
  return new PromiseCancellable((resolve, reject) => {
    p.then((v) => { ... });
  });
}

This would explain why the default cancellation doesn't do anything.
Because we cannot associate the signal to anything here, we would have to add the default functionality here.
Now we could use call PromiseCancellable.from here to reassociate the default handler. But again this seems like a waste of resources, we already have a PromiseCancellable, it's just a raw one, and not one produced by then.
This seems kind of flaky unfortunately. Perhaps this would have worked better if the constructor itself already associate the default signal handler. I just didn't do that originally thinking the signal handler is already custom as part of the executor. But this is making me rethink things.

The reason it is flaky is that we won't really know which of the static methods vs instance methods are going to be using new vs then or some other method. They could be implemented in a variety of ways.
Now that then implies from, this means that if it is using new, we have to associate the default signal handler, but if it is using then, we do not, because it is already associated.
So far it seems like all the static methods will be using the new method, whereas all the instance methods EXCEPT then itself will be using then.
So we could fix this right now by associating the default signal handler for all static methods, while not doing so for catch and finally.
Alternatively IF the default signal handler was done in the constructor. Then if the symbol species was removed, we would end up with all static methods having the signal handler properly done, and all instance methods rely on then, which itself ends up using the constructor as well and having its default signal handler associated too.
Then each method just needs to override the signal handler if a custom signal handler is passed in, or override the abort controller if it is passed in.

We could reserve the usage of onabort for the new constructor, so that the new usage can still override the signal handler by doing signal.onabort = null or attaching it on onabort... Either way this is an additional statement to "disable" the default early rejection.
We could also make the new PromiseCancellable take the PromiseCancellableController, such that if it is undefined, it can default on the default handler.
This still works...
const pC = PromiseCancellable(executor, (signal) => {
  console.log(pC);
});

Although if you are supplying a custom signal handler, it's quite likely you do not want to do early rejection of the pC.
However how would one do both early rejection AND something else. I guess that's where you can use both the executor and the signal handler.
new PromiseCancellable((resolve, reject, signal) => {
  signal.onabort = () => {
    reject(signal.reason);
    // do other stuff
  };
});

Actually I found that I couldn't do this cleanly because of the case where a PromiseCancellable is simply constructed around something taking the signal. It would only work if we did signal.onabort = null; but this just looks wrong.
On the other hand, the current way with the fix works, but perhaps only for nodejs, if other implementations of promises worked differently this could cause a problem. I'm not sure, will require testing later in the browser to see.

Another idea, we put a Proxy around the AbortSignal. This way we can can intercept the addEventListener function or onabort property.
If it is set, it takes precedence. If it is not set, we will default to early rejection. This way early rejection is set by default. One would argue that if you really want a PromiseCancellable where the cancel does nothing, just add an empty handler to the signal.
This then unifies it under the constructor, and everything relies on the constructor now.

Here's an example of using proxies:
const abortController = new AbortController();

let signalHandled = false;
const signal = new Proxy(abortController.signal, {
  get(target, prop, receiver) {
    if (prop === 'addEventListener') {
      return function addEventListener(...args) {
        signalHandled = true;
        return target[prop].apply(this, args);
      };
    } else {
      return Reflect.get(target, prop, receiver);
    }
  },
  set(target, prop, value) {
    if (prop === 'onabort') {
      signalHandled = true;
    }
    return Reflect.set(target, prop, value);
  }
});

// These 2 do not activate the `signalHandled`
signal.onabort;
signal.addEventListener;

signal.onabort = null;
signal.addEventListener('abort', () => {
});

console.log(signalHandled);

This ensures that we know if the signal has been handled. In this way it is possible to enforce that PromiseCancellable always has its signal handled at some point. And if users really want to prevent anything from happening, they just need to do signal.onabort = null.

This is now done.

Note that I switched to using constructors for centralising the default signal handler logic. Now promise cancellable will always have a default signal handler. The only way to prevent it is to supply your own custom handler, or within the executor, set the onabort to null.
new PromiseCancellable((resolve, reject, signal) => {
  signal.onabort = null;
});

Doing so will prevent the default signal handler.
I thought I could remove the symbol species and rely on the super.then using the PromiseCancellable constructor, however this broke the delayed cancellation tests. Couldn't figure out why, so I kept it as is, and all tests pass now.


RazorViewEngineOptions.ParseOptions are hard to use

Trying to set the C# lang version that Razor uses via the RazorViewEngineOptions.ParseOptions is a less-than-optimal experience right now, as the Roslyn types are immutable.
The following doesn't compile as ParseOptions.LanguageVersion is not settable:
services.AddMvc().AddRazorOptions(options =>
                options.ParseOptions.LanguageVersion = LanguageVersion.CSharp7);

The IntelliSense leads you down a path where this looks like what you should do, but it doesn't work, because WithLanguageVersion returns a new instance of ParseOptions which gets lost to the ether:
services.AddMvc().AddRazorOptions(options =>
                options.ParseOptions.WithLanguageVersion(LanguageVersion.CSharp7));

This is actually what you need to do:
services.AddMvc().AddRazorOptions(options =>
                options.ParseOptions = new CSharpParseOptions(LanguageVersion.CSharp7));

Or probably more correctly:
services.AddMvc().AddRazorOptions(options =>
                options.ParseOptions = options.ParseOptions.WithLanguageVersion(LanguageVersion.CSharp7));

Not really sure we can do anything about this given the types are designed this way.

ref return ParseOptions ?

IMO we should not support/expose this at all. It's confusing and we don't really do what it says.
Here are all 20 reasons why:

Changing this setting doesn't help because Razor doesn't respect it
No I'm not referring to how we don't currently have support for C#7
What I mean is that when we do have support for C#7, Razor's understanding of C# won't revert back to C#6
This may do the right thing, or may not, but we shouldn't advertise it if we're not going to test it
Putting a Roslyn type like this in DI via options causes a LOT of things to be jitted even when we don't compile anything
Putting a Roslyn type like this in DI via options causes a LOT of things to be jitted even when we don't compile anything
Putting a Roslyn type like this in DI via options causes a LOT of things to be jitted even when we don't compile anything
Putting a Roslyn type like this in DI via options causes a LOT of things to be jitted even when we don't compile anything
Putting a Roslyn type like this in DI via options causes a LOT of things to be jitted even when we don't compile anything
Putting a Roslyn type like this in DI via options causes a LOT of things to be jitted even when we don't compile anything
Putting a Roslyn type like this in DI via options causes a LOT of things to be jitted even when we don't compile anything
Putting a Roslyn type like this in DI via options causes a LOT of things to be jitted even when we don't compile anything
Putting a Roslyn type like this in DI via options causes a LOT of things to be jitted even when we don't compile anything
Putting a Roslyn type like this in DI via options causes a LOT of things to be jitted even when we don't compile anything
Putting a Roslyn type like this in DI via options causes a LOT of things to be jitted even when we don't compile anything
Putting a Roslyn type like this in DI via options causes a LOT of things to be jitted even when we don't compile anything
Putting a Roslyn type like this in DI via options causes a LOT of things to be jitted even when we don't compile anything
Putting a Roslyn type like this in DI via options causes a LOT of things to be jitted even when we don't compile anything
Putting a Roslyn type like this in DI via options causes a LOT of things to be jitted even when we don't compile anything
Putting a Roslyn type like this in DI via options causes a LOT of things to be jitted even when we don't compile anything

@davidfowl who asked for this feature. I think you're right - we should probably have a select set of things exposed on view engine options (additional references for instance). We could have extensibility \ replacability around things like CSharpCompiler in the event the user actually wants to meddle with the compilation rather than having it exposed on such a publicly visible type.

@rynowak where did we land on this?

I haven't started this yet. If you think you can get it done today that would be a load of of my mind 👍

Sure, I can take over this.


fix(ConnectionString): missing name for getBoolean

Description
in follow example:
const newSet = new Set(['false',false].map(getBoolean));

console.log(newSet.size === 1) // false

const newSetB = new Set(['false',false].map(getBoolean.bind(null,'tls')));

console.log(newSetB.size === 1) // true

the newSet would contain two values false, true, because getBoolean is checking index 0 and 1 instead of values.
this bug makes  mongodb connection is always failed.
What changed?
set name to getBoolean
Are there any files to ignore?

I incorporated the fix in a slightly different form in the linked PR #2979 to make the driver validation of these options more spec compliant. Thank you again for calling this out.


Derive OpenAPI type schema from pyspec

Thanks to ethereum/consensus-specs/pull/3506 we have canonical mapping of SSZ -> YAML, so it is possible to derive OpenAPI schemas for all the spec types.
Since the vast majority of API types are spec types auto-deriving the types would reduce future fork development and maintenance. Plus it paves the way to SSZ-ing most API routes.
We can define API-only extra types in SSZ syntax, such as:
// extra_types/bellatrix.md

## Containers

### Registration

From the Builder API specification

```python
class ValidatorRegistration(Container):
    fee_recipient: ExecutionAddress  # Address to receive fees from the block
    gas_limit: uint64  # Preferred gas limit of validator
    timestamp: uint64  # Unix timestamp of registration
    pubkey: BLSPubkey  # BLS public key of validator

class SignedValidatorRegistration(Container):
    message: ValidatorRegistration
    signature: BLSSignature

Current status
I've created a simple python script to translate the pyspec to OpenAPI schemas https://pypi.org/project/pyspec2openapi. Here is a demo integration into the beacon-APIs repo https://github.com/dapplion/eth2.0-APIs/pull/1
The diff is quite large since it merges all types into a single output file.
One can compare the bundled outputs with
git checkout master
swagger-cli bundle ./beacon-node-oapi.yaml -r -t yaml -o ./deploy/beacon-node-oapi-master.yaml
git checkout dapplion/derive-spec-types
swagger-cli bundle ./beacon-node-oapi.yaml -r -t yaml -o ./deploy/beacon-node-oapi-derive.yaml
diff -l -u deploy/beacon-node-oapi-master.yaml deploy/beacon-node-oapi-derive.yaml | colordiff | more -R

The bundled diff is also quite substantial due to some key characteristics

Usage of allOf to define block types. Generated spec does not de-duplicate keys, resulting in less indentations

  BeaconBlock:
    allOf:
      - $ref: '.../BeaconBlockCommon'
      - $ref: '#/Deneb/BeaconBlockBody'

Lots of comments per-property and per type not present in the generated specs
https://github.com/ethereum/beacon-APIs/blob/4882aa0803b622b75bab286b285599d70b7a2429/types/block.yaml#L24

Next steps

[ ] Get buy-in from repo mantainer's, or how could I tune this to align with your prefered DX

If yes, decide how to merge

One big diff PR switching directly to generated types, a-la https://github.com/dapplion/eth2.0-APIs/pull/1
Big diff PR merging existing existing schemas into a single spec.yaml file, then replace with generated schemas for a clearer diff
Modify current master branch to align with generated output (remove block allOf usage, remove comments), then merge a smaller diff PR switching to generated types

ooh, great stuff! :+1: from me, was hoping it would lead to this.

It would be good to maintain the script either in this repo or another repo under ethereum. I feel uneasy about one person maintaining it privately (not from a security perspective, but because of bus factor == 1)

It would be good to maintain the script either in this repo or another repo under ethereum. I feel uneasy about one person maintaining it privately (not from a security perspective, but because of bus factor == 1)

Sounds good, will look how to integrate it without polluting this repo too much. @rolfyone thoughts?

Can we have the conversion script in this repo or in consensus-specs and just run it during build or something? or produce something in consensus-specs as part of that build that we then ingest would be ideal maybe?
It looks like we're stripping out a bunch of types, can we just ingest it and ignore the types we dont need or something? I like the direction, agree polluting this repo would be less than ideal

Can we have the conversion script in this repo or in consensus-specs and just run it during build or something? or produce something in consensus-specs as part of that build that we then ingest would be ideal maybe?

I don't think we should pollute the spec tests, since this is purely derivative it should leave in this repo

can we just ingest it and ignore the types we dont need or something?

There may be some types that are un-used but looks like the minority. For that we should either maintain and opt-in list or an exclusion list. Is it worth the effort to have a few less lines?

If we can just ingest the spec and it works, then the extra objects are pretty un-important, does make it easy when we add endpoints... The only thing would be if people get annoyed at unused objects if they cause issues in code generation maybe?

https://redocly.com/docs/cli/commands/bundle/ offers a --remove-unused-components option that should take care of unused objects.
Note that we are currently not using redocly for bundling but swagger-cli. swagger-cli is deprecated and recommends migrating to redocly


Aircraft taxiing to runways not approved by SID

I was reading through issue #324 and thought this was handled by the complex SIDs update, so maybe I'm coding something incorrectly but the image below should explain it; aircraft who have filed the SKORR3 departure should depart on rwy 31L/31R according to the chart (and I've omitted the other runways in the code) however pilots are still taxing to other runways when given the unspecific "taxi" clearance.  I thought that the game did not allow aircraft to use a SID if it was not available for the default runway assignment?

Yeah, I think this was intended, and was somewhat lazy on my part because I didn't want to have to figure out the hard part... If they are given their IFR clearance, they should file for a SID that will be valid from the runways most likely to be active. And if ATC uses different runways, then we would tell the aircraft their route is invalid and to go talk to clearance delivery again, who would assign them a SID that works with the runways we're actually using. For that, we would have to add a rerouting command. And it should also probably alert the user upon taxi that "my filed SID is invalid from Runway XX" to guide us to reroute him.
I'm pretty sure the current behavior is this: if the SID doesn't have a "rwy" property for the runway the aircraft is using for takeoff, skip ahead and assign the "body" (if available) and "transition" (if available) sections so they at least do something reasonable.
But of course, the eventual intent was to make it so that they cannot take off in the pictures situation without some amendment, either to their SID, or perhaps just skip the assigned SID altogether by saying BAW127 dct YNKEE fh220 takeoff. That would be a common real world workaround too. Basically, I just never fully implemented this.


Master optimum-intel for llava

Looks like safe to merge?

The hope was that if would fix llava-next output. But it didn't. So I don't care if all the models are tested with a fork or some of them are tested with upstream.


chore: update Github actions

What it solves
Update Github actions to use latest dependencies removing deprecation warning

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 73.632%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
476

Relevant Lines:
597

💛  - Coveralls


Cargo output is not printed if there is an error

If there is an error in cargo check command it is not printed when verbosity is set to 5
Steps to reproduce

Create a rust project  using command init -t rust --name some-test -v 5 -p ./
The cargo check fails due to error.

DEBUG: 2024/01/28 19:37:41 Executing Command: /usr/bin/git clone --single-branch -b rust --depth 1 https://github.com/csd1100/templates/
DEBUG: 2024/01/28 19:37:43 Output of command /usr/bin/git clone --single-branch -b rust --depth 1 https://github.com/csd1100/templates/:
Cloning into 'templates'...
DEBUG: 2024/01/28 19:37:43 Executing Command: /usr/bin/git init
DEBUG: 2024/01/28 19:37:43 Output of command /usr/bin/git init:
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: 	git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: 	git branch -m <name>
Initialized empty Git repository in /tmp/init-3873798466/templates/.git/
DEBUG: 2024/01/28 19:37:43 Executing Command: /home/user/.cargo/bin/cargo fetch
DEBUG: 2024/01/28 19:37:43 Output of command /home/user/.cargo/bin/cargo fetch:
DEBUG: 2024/01/28 19:37:43 Executing Command: /home/user/.cargo/bin/cargo check
ERROR: 2024/01/28 19:37:43 Error while running sync, Err:exit status 101

Expected Result
The output shows cargo output
Actual Result
There is no output of command printed

There was no logger message to print STDERR in case cmd.CombinedOutput() returned an error.


Enable InstaClick by default

It will be the new default for Turbo 8. You can always opt out by setting <meta name="turbo-prefetch" content="false"> in the head of your HTML.
To help older apps with the transition we don't prefetch any links that contain the data-remote, data-behavior, data-method or data-confirm attributes. These are used by UJS in older apps to trigger unsafe behavior on links. More modern apps tend to implement these as <button>s instead.
//cc @dhh @davidalejandroaguilar

is it possible to skip also the  with data-turbo-stream="true" ?

@navidemad if you always want to skip any link that has data-turbo-stream="true" you could also do so in a turbo:before-prefetch event handler, along the line of this example that @afcapel posted earlier.


Markdown to markdown transformation

It seems it's only possible to render to html at the moment.
It would be useful to save intermediate processing in markdown format in such a way that
this could later be used by different rendering tools.
Is there a clear path to implement this? Any pointers?

It produces an event stream, which can be saved or otherwise processed. This event stream is similar to the AST produced by cmark, but is an iterator rather than an allocated-in-memory tree. I think anything more than that is beyond the scope of this crate. Maybe I just don't understand what you're asking for.

It produces an event stream, which can be saved or otherwise processed. This event stream is similar to the AST produced by cmark, but is an iterator rather than an allocated-in-memory tree.

I understand how it currently works (I think).
What I am looking for is a way to use the library to parse markdown and output markdown, as that would be useful for "markdown tools" that work enriching or transforming markdown files.

I think anything more than that is beyond the scope of this crate

Maybe so, this is more of a "is there a easier way?" type of question: do I need to implement another renderer for the event stream from ground up or could I reuse something existing?

Ah, you're talking about markdown-to-markdown transformation. That's possibly interesting.

There's this project which seems to do what you are looking for. It does still rely on version 0.2 of pulldown-cmark, so it won't be compliant with the latest version of CommonMark. It shouldn't be too hard to update it to use the most recent version though.
Closing this for now as this seems to be solvable out-of-crate. Happy to reopen on new insights.


ImportError: DLL load failed while importing _pywrap_tensorflow_internal: The specified module could not be found.

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):-Windows 10
TensorFlow installed from (source or binary): Pip Install Tensorflow
TensorFlow version:2.2.0
Python version:3.8.2 64 bit
Installed using virtualenv? pip? conda?: PIP
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version: i=I don't have Nvidia Graphic Card
GPU model and memory: Intel R HD family (2160 MB)
-Laptop- Hp EliteBook 8 GB ram ; core i5

Describe the problem
I am having a problem as Import error. Here is the stack code.
Traceback (most recent call last):
File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\imp.py", line 242, in load_module
return load_dynamic(name, filename, file)
File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\imp.py", line 342, in load_dynamic
return _load(spec)
ImportError: DLL load failed while importing _pywrap_tensorflow_internal: The specified module could not be found.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "f:/MACHINE LEARNING/chatbot/main.py", line 9, in
import tensorflow
File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow_init_.py", line 41, in
from tensorflow.python.tools import module_util as module_util
File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python_init.py", line 50, in
from tensorflow.python import pywrap_tensorflow
File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 69, in
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\imp.py", line 242, in load_module
return load_dynamic(name, filename, file)
File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\imp.py", line 342, in load_dynamic
return _load(spec)
ImportError: DLL load failed while importing _pywrap_tensorflow_internal: The specified module could not be found.
It happened only after I just imported TensorFlow. Is it because I don't have a graphic card?
Please Help.

@dhruv-colosus,
Could you please check this comment on a similar issue and let us know if it helps. Thanks!

@dhruv-colosus,
Could you please check this comment on a similar issue and let us know if it helps. Thanks!

Ok, I have avx2 SUPPORT. I already have MSVC 2017 installed. Python is 64 BIT.  I think the problem is with the 2017 version instead of 2019.

I solved my issue by upgrading it to 2019. Thank You.Closing The Issue


Update README.md

add godog. to the second TestMain example to make it runnable

Codecov Report

Merging #111 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master    #111   +/-   ##
======================================
  Coverage    79.7%   79.7%
======================================
  Files          16      16
  Lines        2341    2341
======================================
  Hits         1866    1866
  Misses        360     360
  Partials      115     115

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 272624a...060dc38. Read the comment docs.

indeed, thanks ;)