Move this repo into a shared space and expand into a bigger "framework"

Copy paste from the message I sent a while back
Hey guys,
I added both of you to this group as I think we are all "stakeholders" (if that is the correct word to use), in the potential for an expanded Google Play Music JS Framework that extends on what @twolfson has already done in his Google Music node package.
I have recently made a branch on my player that changes all my JS to use the Google Music node package, but I realized while testing that what I was doing had a few issues
a) My code was still fragmented, some of it used this marvelous API. Whilst the rest was doing everything manually with obscure, unrecognizable selectors
b) I had no control over this framework, if something broke I couldn't push out a fix in a few hours I would have to wait for either @twolfson to fix it or submit a PR and hope it gets merged
c) These fixes still relied on the user updating their application, if for whatever reason they decided not too (some users are stubborn like that) they will not get the fixes...
I would like to propose the following
We should either fork or move the Google Music node repository into a space where people who use it (Me, Radiant Player, Electron Player, Etc.) will have contributor rights to allow one of us to push fixes ASAP. This will allow changes to go out much quicker as it will take only one of us to fix everyone's applications
Instead of depending upon statically released JS files we should depend on a CDN? Or the latest github release? so that updates to this framework, can be pushed independently of a version bump on our players
This proposed framework should maintain the existing functionality of the current Google Music node package but should also have add on packages that allow theming support (I have some funky JS in my repo at the moment that does this) and other things like possibly a central last.fm intergration library?
What do you guys think of this proposal?
Tagging relevant parties
@twolfson @jacobwgillespie @chrismou

First off, thanks for raising this issue. It's good that we are finally trying to unify our efforts with respect to handling the brittle DOM of Google Music.
I will talk to each about points individually.
Fragmented code
With respect to using a common API but it not encompassing the entirety of your app, this is kind of like chasing The Holy Grail. We are different applications that have some overlap but are not duplicates of each other.
For example, Radiant has theming and hides Google notification whereas I choose to avoid the potential brittle-ness and keep the UI more/less the same.
That being said, there is definitely some more room to help each other out (as you say -- last.fm integration would be nice). However, I don't think an overarching framework is the way to do that. Instead, I would prefer more one-off repos (so I can only download/import what I need) and maybe a larger library that allows you to bind them all at once (if that floats your boat).
Common ownership
I'm all for having common ownership. We have been wanting that for a long while (radiant-player/radiant-player-mac#252).
I think that the argument about something breaking without a quick turnaround is a bit of a strawman. When Google Music shifted to Material, we updated our integration within 24 hours.
While I see your concern, I also don't want to maintain a repository that is being cowboy committed (i.e. without a code review process). My suggestion would be something like:

Issue appears
Person A triages it and writes fix
Person A opens PR
If person B sees the PR and is available

Person B performs code review

Otherwisei f person B sees the PR but is unavailable to thoroughly review, person B gives ETA

If ETA is tolerable, then person A waits for full review

Otherwise if person A doesn't hear from anyone within reasonable time (e.g. 24 hours)

Person A can cowboy land changes

I should also mention that npm supports referencing git URLs. This means we can publish our application with a non-published google-music.js.
https://docs.npmjs.com/files/package.json#dependencies
https://docs.npmjs.com/files/package.json#git-urls-as-dependencies
Fixes rely on users updating their applications
While I understand your concern, I think this is attempting to solve an effect of a deeper problem. There's usually a reason for people refusing to update (e.g. nothing telling them about new update, not willing to take time to update). For that, I would suggest finding out why and maybe adopting an auto-updater (e.g. Squirrel -- what's used by Electron):
https://github.com/Squirrel
Additionally, if we were to always point our JS to master, then it will only work until we introducing a breaking change. At some point, our API might change; we might rename a method or change a function signature for ease of use or consistency. We don't want to code ourselves into a corner which will break everyone =/
One alternative would be to have major/minor branches as part of updates (e.g. for a release like 1.2.3, we would update a branch 1.x.x and 1.2.x which point to the same version). That could be used with http://rawgit.com/. I use the same technique on gratipay-badge:
https://github.com/gratipay/gratipay-badge
tl;dr
To summarize everything:

Open to moving to a GitHub organization as long as we have a conservative code review process in place
Opposed to making a catch-all JS repo for all our features, would prefer targeted one-off repos (e.g. low-level bindings to page, library for theming, library for last.fm)
Opposed to always pointing to master and personally prefer checking releases by hand but open to adding major/minor branches

Here's my thoughts:
Fragmented Code
I agree with @twolfson, I like the idea of multiple low-level modules with a common integration module.  IDK if this should be one repo (with multiple modules within it) or multiple repos.
Common ownership
Common ownership is great.  I like the argument for it based on fast turnaround, though I would phrase it more as "bus factor".  As in, the library shouldn't depend on a single developer's time.
I completely agree about cowboy-committed repos.  With Radiant we're doing pretty much what you've described where somebody opens a PR, then another collaborator does a code review, then the original submitter merges after approval.  It's more or less documented at https://github.com/radiant-player/radiant-player-mac/blob/master/CONTRIBUTING.md.
Updating applications
I like the idea of having proper semver and publishing 1.x, 2.x versions that are live-loaded over HTTPS, since then we can push fixes while not breaking backwards compatibility.  Eventually we'd stop pushing fixes to older branches, so at some point users would need to update their apps, and for that I like the emphasis on each individual app implementing an easy / auto updating system.
The org
I'm not sure where you all think this should be moved.  I'd suggest @radiant-player, though my main concern there is currently @kbhomes is the only admin on that org, so any admin-related tasks would have to run through him (bus-factor issue).  I'll see if he's open to adding at least one more admin to the org, which would eliminate that concern.  Or perhaps a new org should be created just for this project...

Hey guys
As I mentioned on a previous conversation, I'm definitely up for more pooling of resources. They way I see it, the more that can be abstracted away to a centralised repository the better, while leaving it up to each individual application to worry about what other features they provide.  A good example is I've just spotted (and PR'd for radiant player) that Google look to have changed their song title element ID from #player-song-title to #currently-playing-title, breaking scrobbling and displaying all song title notifications as being "undefined".  Course there's no way to avoid Google's changes, there's a good chance that the both of you will either a) Have already noticed and updated to reflect this change, or b) Now need to check to see if you're affected.
But yeah, minor tweaks aside, there's definitely scope for some of the suggestions here.  last.fm integration is a biggie for me - I've fixed several issues on radiant player, and I'm exploring the possibility of moving the entire process out to the JS side of the app (right now it calls an Objective-C service).  And I've started work on a POC for last.fm integration for @twolfson's Google Music Electron which, inevitably, will eventually also make it's way into radiant player if @jacobwgillespie's electron branch of Radiant Player ever takes off. It does feels like I'm doing the same thing multiple times over in slightly different ways.
App updates are probably something that should be left to the individual app. Radiant I think uses the sparkle framework, which is pretty seamless. I doubt we'll want to move away from that any time soon.
But one thing we've definitely got down now since moving to an organisation is the peer review stuff.  I recently added a contributing doc (https://github.com/chrismou/radiant-player-mac/blob/master/CONTRIBUTING.md) which we've stuck to fairly religiously so far.  No merges without at least one OK from another contributor, and for biggies I've been looking for at least 2, squashing commits for PRs, no functional changes committed directly to master, etc. It's a process that's worked well so far, and can only get better if there are more active developers involved (which, from what I've seen, all you guys are pretty damn responsive when it comes to fixes/PRs on your respective repos).
So yeah, I feel like there's definitely scope for some sort of org to divvy out some of the repeated jobs to make all the apps more stable, but really it comes down to what we could make generic enough for it to be worth centralising.  google-music.js does what it does really well, last.fm integration would be great but I'd have to think about how this can be made generic enough for it to be worth it. At the end of the day, it's going to involve just pulling in an existing last.fm library (there are several good ones) and just writing a wrapper to interact with the relevant selectors.  Theming could be a biggie, and I'd happily get stuck in with helping develop that, but again, there'd need to be discussion on how this'd be implemented in a way that would be useful to all of us (and any other apps that want to use it).  Radiant's theming is good in some ways, but a bit of a pain in others - like the styles built by the Objective-C side of the app, and requiring a restart to apply the new theme. There's a lot of room for improvement there.
So yeah, the tl;dr - if you guys are up for that discussion, then I'm certainly on board. It'd be nice to get through a week without having to investigate why something's broken and spending what time I have available working on new features!  ;-)

Ha! It does make me laugh at despite being an 8 hour time difference apart, 3 of us have chipped in within an hour of each other. And @MarshallOfSound in over in NZ (god knows what time it is over there).
Github's a brilliant thing. How did we ever live without it.  ;-)

OK, so reading through these insane walls of text (It's actually great to get so much collab so fast) :smile_cat: I can see that their seems to be some agreement on a few key points, namely:
Fragmented Code
The general idea here appears to be to have one core repository for the current google-music.js library, with other module? repositories that when installed will extend on the core library.  Current suggestions for these modular repositories are last.fm intergration and some kind of theming library.
Common Ownership
Here most people seemed concerned with the concept of "cowboy committing", which I too see as a horrendously bad habit to get into.  The common ownership should be more of a response time thing, only two people are required to get a fix into master (PR and reviewer),
Versioning / Updating
I don't think I was quite clear on what I meant about independent releases in my original post :disappointed: But it appears as though you picked up on more or less what I was hoping for anyway :grinning:
Basically the branch idea that @jacobwgillespie talks about it what I would be after, being able to load in the latest dist file from a 3.X.X branch that was tagged with stable for instance would be the perfect workflow for me.
What version and how you fetch it would be entirely up the app though, you might decide to push out static files with each release, or to live on the wild side and fetch straight from master (plz no)....
TLDR
We seem to be on more or less the same page, from here I think we should make a new org and move google-music.js their, I've almost got a POC on how to make google-music.js extensible and I'll link that here when it's ready.
We do however need very strict contributing rules for this org as the "bus factor" @jacobwgillespie mentions works both ways, although it might get a group of people to a place faster... If it crashes we all go down with it so bad code in master has got to be prevented at all costs :+1:

I'd never heard the phrase "bus factor" before, but I like it. Am definitely going to start dropping that into buzzword bingo in meetings at work :laughing:
For the most part, as long as we maintain a decent test suite and don't fall into the trap of :+1:ing potentially breaking changes by eye because they sort of look OK, it should be fine.
Plus once the libraries start getting integrated into the apps we'll at least have several real world examples to test changes against before they get the green light. The feasibility of that should just naturally increase over time.
Anyways, it's 4am over here so I'm off to bed.  Didn't realise the message before was from you! Wooops.
Now I need to check over all the code I wrote before bed. My brain clearly wasn't operating at full capacity. :-P

Alright, since we are all on the same page, it's time to start taking action on these pieces:

Move to a GitHub organization (requires feedback from others)

I don't feel like the radiant-player organization is the best place. How about a new organization like google-music.js? (going to register/squat on it for now)

https://github.com/google-music-js

Add/agree to CONTRIBUTING.md (I will open a PR for this)

Most likely based off of existing CONTRIBUTING.md

Introduce major/minor branches to release process (I will open a PR for this)

As mentioned before, this already exists on gratipay/gratipay-badge -- https://github.com/gratipay/gratipay-badge/blob/2.3.0/package.json#L29-L34
I will take the time to document the release process and document the rawgit.com usage

Addendum to the GitHub organization: Since everyone has different interests with respect to theming, I would suggest we set up different teams with different admins for each of these repos. Admins on each team can create their own repos (e.g. multiple theming ones -- prob one for the integration, maybe another one that is a collection of CSS variations).

Damn, hit a snag with rawgit.com.  There are options like S3 and such but I feel like we are starting to make a mountain out of a molehill =/

@twolfson The github API allows 60 unauthorised requests per hour from a given IP address, surely we won't exceed that many requests just fetching a raw file from github?

This isn't attached to the GitHub API, it's their raw server. If you are unfamiliar with hotlinking to the GitHub raw server, GitHub dislikes it so much that they moved to always sending Content-Type: text/plain to avoid it. As a result, people built things like rawgit.com:
http://rawgit.com/faq#why
We could try using rawgit.com's non-CDN feature but I would recommend against it =/
http://rawgit.com/faq#what-happens-to-jerks

This service seems to fulfill the requirements?
https://github.com/schme16/gitcdn.xyz/blob/master/README.md
On Mon, 14 Dec 2015, 13:34 Todd Wolfson<EMAIL_ADDRESS>wrote:

This isn't attached to the GitHub API, it's their raw server. If you are
unfamiliar with hotlinking to the GitHub raw server, GitHub dislikes it so
much that they moved to always sending Content-Type: text/plain to avoid
it. As a result, people built things like rawgit.com:
http://rawgit.com/faq#why
We could try using rawgit.com's non-CDN feature but I would recommend
against it =/
http://rawgit.com/faq#what-happens-to-jerks
—
Reply to this email directly or view it on GitHub
https://github.com/twolfson/google-music.js/issues/13#issuecomment-164314617
.

Ah, yea. That should work great. Nice find :+1:

I'd put in a vote for cdnjs as the CDN - they have auto-update with npm, which is nice.

Could you link us to the docs on that?

Yep, it's in the README at https://github.com/cdnjs/cdnjs

Sooo the README says it has a 30 hour turnaround time which prob isn't what we want =/
https://github.com/cdnjs/cdnjs/tree/91057ed60af50889bfc710e70e760b680c5bcaf8#enabling-npmrecommended-or-git-auto-update
Additionally, I don't see anything about dynamic servers (e.g. 2.x.x). It seems to be only for git tag which should be reserved for exact semvers (e.g. 2.3.4)

Check out jsDelivr then - looks kinda cool and has some features like version aliasing (which we could use for the 2.x.x versions): https://www.jsdelivr.com/features/jsdelivr-cdn-features

Yea, that should work as well. The README says it operates based on branches:
https://github.com/jsdelivr/jsdelivr/tree/4cc2747e3164725b672005e4f8b0c37f85f7e253#version-aliasing
So at the end of the day, gitcdn.xyz and jsDelivr would work identically but jsDelivr requires a little bit more setup.
One issue I am having difficulty finding info on is how long jsDelivr caches info for. gitcdn.xyz had 2 hours which is more/less a good TTL for us (prob 5 min to 2 hours is our sweet spot).

Yeah, I would vote that we use something like CDNjs or jsDelivr due to the corporate backing - i.e. it's more likely to stay around.  I think CDNjs is sponsored primarily by CloudFlare, and jsDelivr is sponsored by several CDNs (MaxCDN, CloudFlare, others).  gitcdn.xyz has a note at https://github.com/schme16/gitcdn.xyz/blob/7e2412a05b96389ea78d455203080a3e81dd49ae/README.md#acknowledgments--background saying "As this service is free, I make no guarantees about Up-time nor do I have anything resembling an SLA."  Seems more like a hobby project to me.
For jsDelivr, here's an issue that provides some insight (and it looks like it might be possible to get access to a "purge" API that's still in development): jsdelivr/jsdelivr#6913

From the linked jsdelivr/jsdelivr#6913, there is an issue https://github.com/jsdelivr/jsdelivr/issues/5056 which has a maintainer stating the TTL for aliases is 7 days =/
https://github.com/jsdelivr/jsdelivr/issues/5056#issuecomment-108898293
If we want to use jsDelivr, we will definitely have to get access to their purge implementation so we can cache bust upon deployment.
My recommendation for now would to be move forward with gitcdn.xyz (even though its a bit experimental) but it will move us along to a better place (e.g. have someone to proof of concept with for semver branches for those that want to use them). Then, on the side we can open an issue and see if jsDelivr will grant us access to their purge system.

hey guys,
So I managed to get us an open source license to use browserstacks systems to CI this library (woo hoo), if you guys want to be added to the team that has access to the browserstack API just send me a message in the gitter room I added you to, or my email address.
/cc @twolfson @jacobwgillespie @chrismou

Yo, but late to the party here, but re: org names, would it make more sense to use something a bit more generic?  Google Music Utilities? Google Music Library/Libraries?
Especially since there's a good chance the repos won't be 100% JS. Plus I know the pain of working for a company that named after a language - despite having "Flash" in our company's name, we're now about 6 months off dropping Flash support completely in favour of HTML5.  Wooops.   ;-)

I'm actually wondering if it should be named something else that doesn't contain "Google Play" in the name - if our library gets popular, I'm envisioning an email from the Google trademark team...  (I believe Radiant went through something of that nature in the past)

Using a name like "gmusic" is a simple way of getting round that. I assume that's how there's apps like GMaps exist in app stores.

I'm not sure we should ever have anything non-JS. If we are building something that is a non-JS lib then chances are it could be generalized to not be specific to Google Music (e.g. could be reused for Spotify, YouTube). At that point, I'm not sure it would fit into the organization.
With respect to gmusic, someone is squatting on that in npm so we would have to use something like gmusic.js =/
https://www.npmjs.com/package/gmusic
https://www.npmjs.com/package/gmusic.js

@twolfson Wasn't it you that mentioned theme repos? That's going to be pretty much exclusively CSS I would have thought. And they'd be specific to overriding styles on the google music stylesheet. They could be used on literally any Google music project, not exclusively JS based.

Ah, good point :+1:

The conversation seems to have halted. I'm going to break it down into 2 issues --

Decide on name for new organization (keeping it on this thread)
Decide on how to (if at all) handle hosted floating major/minor releases (moving to a new thread)

RE: new name
Google Music Framework
has my vote

I have create a new issue for the floating major/minor releases in #22

I agree with @chrismou's concerns about trademark infringement. On another project, we had to deal with the painful process as well (e.g. tight deadline requested by company, need to update all docs, update all CI). https://github.com/plaidchat/plaidchat/issues/79
As a result, my vote is for gmusic

:+1: for gmusic, the org name is already take on GH.  gmusic tools? gmusic library? gmusic framework?
Then repo names could be gmusic.js, etc

Ah, damn =(
How about gmusic-utils?
I don't think library nor framework is exactly fitting for the organization since it's more of an aggregation than an overarching construction.

Either gmusic-utils or gmusic-tools works for me. Would probably sway towards tools at a push because it's easier to say!

Heh, a similar rationalization for me about gmusic-utils is that "utils" is more prevalent (e.g. suitcss' utils, GNU's coreutils)
https://github.com/suitcss?utf8=✓&query=utils
http://www.gnu.org/software/coreutils/coreutils.html

Leaning towards utils myself :+1:
gmusic-utils

Alright, I will wait until 12PM CST on Sunday for @jacobwgillespie for submit his vote.

In any event, here's a checklist derived from what we did on plaidchat:

[ ] Rename organization
[ ] Rename repo
[ ] Invite collaborators to team
[ ] Update README/docs references
[ ] Add notice "not created by, affiliated with, or supported by Google Inc."
[ ] Publish latest version to npm
[ ] Add collaborators to npm
[ ] Publish notice to google-music README on npm about rename
[ ] Create test PR to verify Travis CI hooks are working

I vote gmusic-utils

Woot, I will take a shot at performing the rename some time this weekend =)

Going to grab/squat on that organization ASAP now though ;)

Alright, organization created and sent out invites :tada: https://www.grubhub.com/

Heh, wrong link from another conversation

Was like "already got lunch, but..."  :)
:+1:

Alright, going to perform rename now. We didn't discuss casing at all but here are my thoughts:

Organization formal name: gmusic-utils (all lower case)
Repo formal name: gmusic.js (all lower case)
Repo constructor/docs GMusic (proper case)

@MarshallOfSound There's a task on the checklist that's relevant to you. Can you update the name of the BrowserStack account? IIRC, it was something like "Google Music...". It should probably be gmusic-utils or gmusic.js now

I'm going to go create a meta repository now so we can discuss meta topics like avatars for the organization

@twolfson I didn't actually name that team, BrowserStack did.  I'll submit a support ticket to get them to rename it, not sure if I can do it myself

Oh, weird o_o Thanks =)

Woot, will check off that item on the list =)

Alright, now that we are live on npm in 4.0.0 I can add contributors. What are your npm handles?

Deprecation notice is posted on https://www.npmjs.com/package/google-music

@twolfson My npm handle is marshallofsound
https://www.npmjs.com/~marshallofsound

Alright, you are now an owner on npm:
https://www.npmjs.com/package/gmusic.js

Going to close this PR pre-emptively since all items but #25 have been taken care of =)


Adding RailsCasts link

Adding RailsCasts link, it is not free courses site but it contains free screencast. Therefore adding it here. Please check and accept the Pull request.

Super interesting, I never heard about this


Webcrypto

this removes the dependency on crypto.js and uses the webcrypto API. I haven't updated the CLI with the new logic, but I wanted to open this PR with what I had so far. #20

It looks like you can't use a the web crypto API in a node module. So I don't think we can remove the crypto.js dependency from the CLI.

@tarpdalton can't you continue to use crypto-js in the node module, but just change the code in the template HTML file to use the web crypto API (and fallback to asynchronously loading crypto-js if needed?)

@epicfaace yeah we could do that. We would just have to make sure the something encrypted by crypto-js can be decrypted by webcrypto correctly

I guess this can be revisited now since Node.js 15 has a Webcrypto module now. Current stable Node.js version is 16.
https://nodejs.org/api/webcrypto.html
Also there is this package: https://www.npmjs.com/package/node-webcrypto-ossl

thats cool. yeah we should use the webcrypto api in the node, and then crypto-js can be fully removed

i got the cli working with node webcrypto api. crypto-js is removed. I don't really use this lib so I can't test it out, but it worked for some simple things. Someone should test it out before it merges

We now finally rely on WebCrypto in StatiCrypt! The code was merged in #164 , it's pretty noisy due to wanting to be as compatible as possible with 2.x custom templates, but it'll get much cleaner in 3.x (the WIP branch is already here).
Thank you very much for opening and updating this PR over the years, it's been a good sanity check for me to read your approach here and useful for the final implementation. As a (very small) token of gratitude, I added your name in the "Thank you! :pray:" README section. Thanks!


[PA-699] Fix broken links and add text for users

TICKET
Description

This PR address engineering support tickets ESN-253 and ESN-289.
For ESN-253 we're adding text to the issues tab and the add issue form to inform users that org admins will be notified of new issues. The added text is "Once an issue has been reported an administrator will be notified. Please check back for updates and resolutions.". This text should only appear if the following has been set in the ini ckanext.issues.send_email_notifications = true.
For ESN-289 we're fixing broken links so that they direct users to the linked issue.
Test Procedure

Checkout this PR and install the issues extension by adding issues to the plugin list
Go to a dataset and click on the issues tab.
Check that there is no text  at the bottom.
Add ckanext.issues.send_email_notifications = true to the ini, restart CKAN, and check that the text Once an issue ... is present now.
Add a new issue, leave a comment, and go back to the dataset issue tab.
Click on the comment link and check that it directs you to the issue.

Approval Criteria

LGTM


sap.m.Carousel: forward loop impossible if "visiblePagesCount" > 1

URL (minimal example if possible):
https://openui5nightly.hana.ondemand.com/test-resources/sap/m/Carousel.html
Steps to reproduce the problem:

Open the above sample page. The loop property of the Carousel is already enabled.
Change the value in Number of pages to be shown in Carousel's visible area below to something higher than 1.
Try to loop forwards.

What is the expected result?
Looping with multiple visible pages works.
What happens instead?
It doesn't
Any other information? (attach screenshot if possible)
Backward looping works.

Hi,
Thank you for sharing this finding. I've created an internal incident<PHONE_NUMBER>. The status of the issue will be updated here in GitHub.
Regards,
Nikolay Hristov

Hello @boghyon
When "visiblePagesCount" is set to something higher than 1, looping shouldn't be possible. This is documented here https://sapui5.hana.ondemand.com/#/api/sap.m.CarouselLayout in the description of this property. However, until now looping was possible backwards, but this is fixed with 7ccb18e
Regards,
Petar

I see, it was the other way around :) thanks for clarifying and fixing it!


Decompression via ZlibStream for large message doesn't work

The below test triggers the following exception in Decompress step for a message bigger than 131072 letters on latest nuget versions. Messages with length less or equal to 131072  letters are decompressed fine. I've checked '0.29.0', '0.28.0', '0.27.0':
Message:
  SharpCompress.Compressors.Deflate.ZlibException : Bad state (incorrect data check)

Stack Trace:
  InflateManager.Inflate(FlushType flush)
  ZlibBaseStream.Read(Byte[] buffer, Int32 offset, Int32 count)
  Stream.CopyTo(Stream destination, Int32 bufferSize)
  CompressorsTests.Decompress(Stream input, Stream output) line 135
  CompressorsTests.Zlib_should_read_the_previously_written_message() line 104

but it works fine for versions <= 0.26.0
    [Fact]
    public void Zlib_should_read_the_previously_written_message()
    {
        var message = new string('a', 131073);  // 131073 causes the failure, but 131072 (-1) doesn't
        var bytes = Encoding.ASCII.GetBytes(message);

        using (var inputStream = new MemoryStream(bytes))
        {
            using (var compressedStream = new MemoryStream())
            using (var byteBufferStream = new BufferedStream(inputStream)) // System.IO
            {
                Compress(byteBufferStream, compressedStream, compressionLevel: 1);
                compressedStream.Position = 0;

                using (var decompressedStream = new MemoryStream())
                {
                    Decompress(compressedStream, decompressedStream);

                    byteBufferStream.Position = 0;
                    var result = Encoding.ASCII.GetString(GetBytes(byteBufferStream));
                    result.Should().Be(message);
                }
            }
        }
    }

    public void Compress(Stream input, Stream output, int compressionLevel)
    {
        using (var zlibStream = new ZlibStream(new NonDisposingStream(output), CompressionMode.Compress, (CompressionLevel)compressionLevel))
        {
            zlibStream.FlushMode = FlushType.Sync;
            input.CopyTo(zlibStream);
        }
    }

    public void Decompress(Stream input, Stream output)
    {
        using (var zlibStream = new ZlibStream(new NonDisposingStream(input), CompressionMode.Decompress))
        {
            zlibStream.CopyTo(output);
        }
    }

    byte[] GetBytes(BufferedStream stream)
    {
        byte[] bytes = new byte[stream.Length];
        stream.Read(bytes, 0, (int)stream.Length);
        return bytes;
    }

Is there any ETA? We use MongoDB Driver that uses this library and there is a vulnerability shown in the reports => it is PITA for the companies which have strict compliance rules. MongoDB can not update it because of this bug.
Thanks!

The "vulnerability" is when the file pathing code is being used.  Would the driver be using that?
I don't have an ETA on this because I'm not actually working on it and there is no PR to merge.
At a guess, iit must be something in the 0.27 release that is causing it: https://github.com/adamhathcock/sharpcompress/releases/tag/0.27
I'd put that test in and start reverting/testing the changes from the 0.27 and see what happens.

Any update?

Sorry no.  I need help

First broken commit.
The solution (at least to fix the above test) is reverting this line (and related nested class). Hopefully this will help.

I actually started with my own zlib by actually using the native version for compression using C# 10 features.
The pull requests list one which I plan to merge soon that replaces the stream with a public static class that uses spans. https://github.com/Elskom/ZlibSharp/pulls/

In more details, the above test works without this condition that returns 1 instead of repeatedly<PHONE_NUMBER>

I've made a PR with the test and identified fix: https://github.com/adamhathcock/sharpcompress/pull/624
Eyeball it and I'll merge it and push to nuget

I actually started with my own zlib by actually using the native version for compression using C# 10 features.
This pull request which I plan to merge soon that replaces the stream with a public static class that uses spans.

If you're interested in porting your zlib into here and possibly helping reworking other algorithms to be Span/Async, let me know.  I started a branch months ago but time got away from me again.  My blocker was my lack of knowledge of the algorithms so doing span/async on them broke and I could not figure why

Should be on nuget soon https://www.nuget.org/packages/SharpCompress/0.30.1

I actually started with my own zlib by actually using the native version for compression using C# 10 features.
This pull request which I plan to merge soon that replaces the stream with a public static class that uses spans.

If you're interested in porting your zlib into here and possibly helping reworking other algorithms to be Span/Async, let me know. I started a branch months ago but time got away from me again. My blocker was my lack of knowledge of the algorithms so doing span/async on them broke and I could not figure why

Also there is another issue as well too on providing a zlib dll for the following RIDs:

win-x86
win-x64
win-arm
win-arm64
linux-x86
linux-x64
linux-arm
linux-arm64
osx-x86
osx-arm64

Currently on linux it uses the system provided zlib (which I do not know if it's 1.2.11 for all linux installs and I do hardcode it to version 1.2.11), For MacOS there is a system provided zlib but I do not know where to locate it so I expect them to install it with homebrew currently. As for Windows, I have no clue what to do and currently expect them to basically compile zlib themselves.
The only thing I can think of it somehow on whatever OS is used to compile to somehow cross compile zlib to all of those RIDs and then package them as native dependencies on the nuget package.
Alternatively I could look into if I can change mny code slightly to use the built in clrcompression library and then not worry about zlib no more but it involves looking at the code to that library that is built into the .NET default runtime.

it works for us, closing this issue then


Update for pyproject.toml-style

With the uptick in pyproject.toml usage, update library without setup.py implementation.

Just came in after a long conversation with some other maintainers hoping to do some automated calver style without a setup.py

Unfortunately this library is basically just a setuptools plugin, so it doesn't have much use in a world without setup.py, mainly because PEP 643 says Version cannot be dynamic.
I think my preference would be to make https://pypi.org/project/bump/ take a --calver flag that just sets the version it to the current date, which could be used prior to building a distribution.


Add some common patterns of failure caused by model rollouts

Hi @xLaszlo,
here we go with the first issue 😃
In the following article https://www.oreilly.com/content/a-brief-guide-to-running-ml-systems-in-production/ you will find a section about common patterns of failure caused by model rollouts.
The whole article is though an interesting read (googles SRE perspective on running ML systems in production).
Kind regards,
Larysa

in


Deploying Kubeflow on Azure is not possible

The GitHub-Repository https://github.com/Azure/kubeflow-aks/ uses the submodule https://github.com/Azure/AKS-Construction/tree/813928a843acd53833286b51d16d7fc6e59ebd6a which is 6 months old. There, the kuberenetes version 1.24.9 is required, see https://github.com/Azure/AKS-Construction/blob/813928a843acd53833286b51d16d7fc6e59ebd6a/bicep/main.bicep#L835. On Azure this kuberenetes version is unsupported. The oldest kubernetes version which is supported is 1.25.6.
Errorlog:
ERROR: {"status":"Failed","error":{"code":"DeploymentFailed","target":"/subscriptions/62ed105d-8213-450b-b6a4-a6ddb768f348/resourceGroups/kubeflow/providers/Microsoft.Resources/deployments/main","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.","details":[{"code":"ResourceDeploymentFailure","target":"/subscriptions/62ed105d-8213-450b-b6a4-a6ddb768f348/resourceGroups/kubeflow/providers/Microsoft.Resources/deployments/aksconstruction","message":"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.","details":[{"code":"DeploymentFailed","target":"/subscriptions/62ed105d-8213-450b-b6a4-a6ddb768f348/resourceGroups/kubeflow/providers/Microsoft.Resources/deployments/aksconstruction","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.","details":[{"code":"BadRequest","target":"/subscriptions/62ed105d-8213-450b-b6a4-a6ddb768f348/resourceGroups/kubeflow/providers/Microsoft.Resources/deployments/aksconstruction","message":"{\r\n  "code": "AgentPoolK8sVersionNotSupported",\r\n  "details": null,\r\n  "message": "Version 1.24.9 is not supported in this region. Please use [az aks get-versions] command to get the supported version list in this region. For more information, please check https://aka.ms/supported-version-list",\r\n  "subcode": ""\r\n}"}]}]}]}}

thanks for your issue @ChristianSqrt4 . working on it

I have the same issue is there is any solution?

closed with #38


[RediSearch] Why is it just experimental?

I was very hyped when I did begin this project to see that Redis has a SearchModule because I think it would be a great lightweight alternative also to Elasticsearch / Opensearch and so on.
While working on the implementation of the RediSearch we did go with JSON Documents as this is the most common usage of it on our side. So we are using JSON and Search module to store our documents.
So where are the problems. Let see we want to index a document like this:
{
    "uuid": "3d3586fe-0416-4572-8ce1-c7b6b9e424ff",
    "title": "Hello World - And other Foo Bar things",
    "tag": "some-tag",
    "description": "A simple description about the article which is searchable."
}

For this documents we create the following Schema:
FT.CREATE news ON JSON PREFIX 1 news SCHEMA $.uuid AS uuid TEXT SORTABLE $.title AS title TEXT $.description AS description TEXT $.tag AS tag TEXT SORTABLE

To write this JSON into our Redis we are using the following code:
JSON.SET news:3d3586fe-0416-4572-8ce1-c7b6b9e424ff $ '{"uuid": "3d3586fe-0416-4572-8ce1-c7b6b9e424ff","title": "Hello World - And other Foo Bar things","tag": "some-tag","description": "A simple description about the article which is searchable."}'

Now as example we want to create a filter on tag or another text field uuid. Here it comes complicated. RediSearch at current state different DIALECTs. If you read over it mostly the created new dialect to provide backward compatibility and still fixing some issues. We decided to go with DIALECT 3 so we are now quering the data:
FT.SEARCH news @uuid:(3d3586fe-0416-4572-8ce1-c7b6b9e424ff) DIALECT 3
FT.SEARCH news @tag:(some-tag) DIALECT 3

No result. Strange. Okay - is a special symbol we need to query the data:
FT.SEARCH news @uuid:(3d3586fe\-0416\-4572\-8ce1\-c7b6b9e424ff) DIALECT 3
FT.SEARCH news @tag:(some\-tag) DIALECT 3

Still no result. So now we remove the document and index it without the - in the tag:
JSON.SET news:3d3586fe-0416-4572-8ce1-c7b6b9e424ff $ '{"uuid": "3d3586fe-0416-4572-8ce1-c7b6b9e424ff","title": "Hello World - And other Foo Bar things","tag": "sometag","description": "A simple description about the article which is searchable."}'

And then search again:
FT.SEARCH news @tag:(sometag) DIALECT 3

And we got now a result. But we manipulated our index data which we didn't want todo as that should contain the - in it and should be returned.
So some reasearch later Redis solution for this problem which is in our case not not usable for us is to escape the value when creating the JSON Document. Yes you read correctly we would need to manipulate our json document which is valid and can be read and write correctly that it works with the -. So lets write a document with:
JSON.SET news:3d3586fe-0416-4572-8ce1-c7b6b9e424ff $ '{"uuid": "3d3586fe\\-0416\\-4572\\-8ce1\\-c7b6b9e424ff","title": "Hello World - And other Foo Bar things","tag": "some\\-tag","description": "A simple description about the article which is searchable."}'

FT.SEARCH news @uuid:(3d3586fe\-0416\-4572\-8ce1\-c7b6b9e424ff) DIALECT 3
FT.SEARCH news @tag:(some\-tag) DIALECT 3

This works but is for us not acceptable solution as reading the document this way:
JSON.GET news:3d3586fe-0416-4572-8ce1-c7b6b9e424ff

Will us return the JSON with \ before the -.
Why the RediSearch would have been a lot of potential this kind of little issue make it sadly not very usable. As filters are common usecase for searches this kind of queries should work without getting strange manipulated JSON back. What the problem is also with \\ as it is not easily can be added to the JSON as a key could also have a - which ends in endless escaping and converting data on all ends.  We also tried to use TAG instead of TEXT type for the field which was mention in some comment but it also did not work there liked expected and ended in the same issue. This is way as current state I can only mark RediSearch @experimental which maybe will end up that we even remove the adapter in the future.
Here the open releated issues on RediSearch repository:

https://github.com/RediSearch/RediSearch/issues/1304
https://github.com/RediSearch/RediSearch/issues/1148

If somebody knows a solution on the Schema side share it with us so we can fix that issue.

After some feedback from @redis and the help of @oshadmi  (Thank you!) https://github.com/RediSearch/RediSearch/issues/1304. I could find a working workaround for this issue. The first step is if a Field is filterablewe create additional.raw` field e.g.:
{
    "uuid": "3d3586fe-0416-4572-8ce1-c7b6b9e424ff",
    "title": "Hello World - And other Foo Bar things",
    "tags": ["some-tag"],
    "tags.raw": ["some-tag"]
}

The .raw fields and the identifier are by default .raw are from type TAG. There is something special about tags and why I thought they also don't work. Because the need to be queried differently then text fields with {} instead of (). This looks like:
-@uuid{3d3586fe\-0416\-4572\-8ce1\-c7b6b9e424ff}
+@uuid(3d3586fe\-0416\-4572\-8ce1\-c7b6b9e424ff)

The schema of the tags.raw we need to escape . this can be achieved in the JSON pointer via:
-$.tags.raw
+$['tags.raw'][*]

So at the end our SCHMEMA looks like this:
FT.CREATE test_complex ON JSON PREFIX 1 test_complex SCHEMA "$['uuid']" AS uuid TAG SORTABLE "$['title']" AS title TEXT "$['article']" AS article TEXT "$['blocks']['text'][*]['title']" AS blocks__text__title TEXT "$['blocks']['text'][*]['description']" AS blocks__text__description TEXT "$['blocks']['embed'][*]['title']" AS blocks__embed__title TEXT "$['footer']['title']" AS footertitle TEXT "$['created.raw']" AS created__raw TAG SORTABLE "$['commentsCount']" AS commentsCount NUMERIC SORTABLE "$['rating']" AS rating NUMERIC SORTABLE "$['comments'][*]['text']" AS commentstext TEXT "$['tags'][*]" AS tags TEXT "$['tags.raw'][*]" AS tags__raw TAG "$['categoryIds'][*]" AS categoryIds NUMERIC

Our document like this:
JSON.SET test_complex:23b30f01-d8fd-4dca-b36a-4710e360a965 $ '{"uuid":"23b30f01-d8fd-4dca-b36a-4710e360a965","title":"New Blog","header":{"image":{"media":1}},"article":"

New Subtitle<\/h2>
A html field with some content<\/p><\/article>","blocks":{"text":[{"_originalIndex":0,"title":"Titel","description":"
Description<\/p>","media":[3,4]},{"_originalIndex":1,"title":"Titel 2"},{"_originalIndex":3,"title":"Titel 4","description":"
Description 4<\/p>","media":[3,4]}],"embed":[{"_originalIndex":2,"title":"Video","media":"https:\/\/www.youtube.com\/watch?v=iYM2zFP3Zn0"}]},"footer":{"title":"New Footer"},"created":"2022-01-24T12:00:00+01:00","commentsCount":2,"rating":3.5,"comments":[{"email":"admin.nonesearchablefield@localhost","text":"Awesome blog!"},{"email":"example.nonesearchablefield@localhost","text":"Like this blog!"}],"tags":["Tech","UI"],"tags.raw":["Tech","UI"],"categoryIds":[1,2]}'

And we can filter values like this via uuid tag field and the tags__raw field:
FT.SEARCH test_complex @uuid:{23b30f01\-d8fd\-4dca\-b36a\-4710e360a965} DIALECT 2
FT.SEARCH test_complex @tags__raw:{UI} DIALECT 3

Thanks Alexander. I've had the same experience. We will prioritize fixing it

@alexander-schranz thanks for the feedback.
One small thing, unless you change the data in the raw fields of the document there is no need to duplicate the data
"$['tags'][*]" AS tags TEXT "$['tags.raw'][*]" AS tags__raw TAG
works as well as
"$['tags'][*]" AS tags TEXT "$['tags'][*]" AS tags__raw TAG

@K-Jo yeah that is true, one of the advantages of redisearch using JSON pointers for the mappings.


Add streamlink config, and utilize yt-dlp in place of youtube-dl

I can keep playing with this some. But right now, both streamlink and yt-dlp appear to be working for me without ads. I've only tested yt-dlp a little bit without ads, as it's late right now.
But effectively:

Add ~/.config/streamlink/config and ~/.config/streamlink/config.twitch which will bypass ads and set best default values for streaming twitch.
Utilize yt-dlp in place of youtube-dl
Adds known headers for yt-dlp to inform twitch that the client is an "ad free client".

So far, both seem to work initially. Skip pre-roll ads when starting a stream and starts very quick. The issue I currently see with both is that when twitch attempts to show a mid-roll ad the stream briefly freezes and so far requires "restarting" (or pressing play, from my client at least) to resume it.
I've been testing from Firefox browser + Apple TV (tvOS) Plex app. I tested from macOS Plex Desktop app as well, but it failed to play with any setting.
Unsure at the moment how to resolve the mid-roll ads breaking the stream. Need to dig more.

I tested a bit more this morning.
Running into it to freezing the stream a bunch and having to restart it. Which unfortunately makes it a pain to use. So I want to continue playing around. Probably shouldn't merge these especially if the freezing doesn't happen for you or anyone else.
I just wanted to use this to drop Turbo subscription, and still be able to watch Twitch without getting hit by ad breaks every 10 minutes.

I want to continue playing around. Probably shouldn't merge these especially if the freezing doesn't happen for you or anyone else.

Sure, no problem! Feel free to play around and see if it gets into a state you'd be comfortable merging. If not, you're of course welcome to keep using your branch, and we can keep it here for reference. Thanks for taking the time to work on this!


Remove session proxy

!test

Refer to this link for build results (access rights to CI server needed):
http://jenkins.2gis.local/job/vmmaster.staging.regression/285/
Test PASSed.

:+1:

:+1:


TextClassifier

I see ImageClassifier in the doc. Is there something like TextClassifier to process text?

+1 would also be interested in auto text classifier

same here!

Thank you for the suggestion.
We are working on this feature now.
It would be online soon.
Thanks.

Suggested Name

TextClassifier
Task Description

Text classifiation
Evaluation Metrics

Accuracy
Benchmark Datasets

Penn Treebank
Reason

NA
Solution

NA
Additional Context

NA

@boyuangong Please write a baseline for the TextClassifier.
It needs to extend from the supervised.py.
And reuse the class of ModelTrainer in utils.py.
Currently, it doesn't need to search for multiple architectures.
Just use a default architecture for now.
We can have a meeting to discuss if you have questions.
Thanks.

Re: TextClassifier benchmarks:
I would suggest starting with a document/sentence leve lclassification model (not word level/many to many). e.g. IMDB sentiment. (Common benchmark for sentiment classification of short documents. Used to benchmark ULMFit amongst others).
https://github.com/keras-team/keras/blob/master/keras/datasets/imdb.py
http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html

@ddofer Thank you for your suggestions.

When do you expect to release the initial TextClassifier model?

@boyuangong  I can help you out with that module if that's okay.

Hi there,
Thanks a lot. I will send the pull request of the newst version tonight when I come back home. The model is pretty much complete now. Only   have some run out of memory problem. It would be great to receive help from you.
-- Best Boyuan Gong

On Sep 15, 2018 at 1:56 PM,  <Muhammad (mailto:notifications@github.com)>  wrote:
@boyuangong (https://github.com/boyuangong)  I can help you out with that module if that's okay.
—
You are receiving this because you were mentioned.
Reply to this email directly,   view it on GitHub (https://github.com/jhfjhfj1/autokeras/issues/35#issuecomment-421616659), or   mute the thread (https://github.com/notifications/unsubscribe-auth/AV8DUPG7diI91B-wkADAb3aMMEJJB4w5ks5ubU1ogaJpZM4VtU_w).

@boyuangong is this live yet? Is there a branch where the code lives currently? Very excited to see your work!

Woooooo!

Hi.
Great, what is the status?
Some questions
-What is the max length of each texts we want to classify? For example can, we proceed an average email ?
-How many classes can we feed in the engine, so it could stay relatively average with not too much false positive ? 20, 50?
I know many old classifier drop a lot in accuracy if there is too many classes.
But recent one like the openSource from Facebook can stand really more. But I don't want to use anything from facebook.
-What is the format to feed for training and then testing ? I didn't see any docs and tutorials yet, is there?
Thanks, great job.

@renaudham Hi, thanks for your interesting in our work. We are preparing the tutorial documents for some modules including the textClassifier. You can directly check this in the new pull request 341. The text tutorial are in text.md.
Also, if you want to try the textClassifier. You can go and check the example in examples/text_cnn.
Please feel free to let me know if you have further questions.


Page Layouts: Divide up Fitness Template

Feature Description
Divide the Fitness template into pages to be individually used as page layouts
Alternatives Considered

Additional Context

Do not alter or remove anything below. The following sections will be managed by moderators only.
Acceptance Criteria

Select the Fitness section in the page layouts library
Each page of the Fitness template should be an individual layout

Implementation Brief

This is automatically handled with how we are importing the template. It is no longer a manual process.

This is automatically handled with how we are importing the template. It is no longer a manual process.


[Bug] Whenever I download for mac it says “BandagedBD” cannot be opened because the developer cannot be verified.

Describe the Bug
Makes me delete it because it says “BandagedBD” cannot be opened because the developer cannot be verified.
To Reproduce
Please fix this
Expected Behavior
I thought it would work because I turned off my firewall
Screenshots
None.
Discord Version
Stable
Additional Context

In macOS Catalina and macOS Mojave, when an app fails to install because it hasn’t been notarized or is from an unidentified developer, it will appear in System Preferences > Security & Privacy, under the General tab. Click Open Anyway to confirm your intent to open or install the app.
The warning prompt reappears, and you can click Open.

https://support.apple.com/en-us/HT202491

In macOS Catalina and macOS Mojave, when an app fails to install because it hasn’t been notarized or is from an unidentified developer, it will appear in System Preferences > Security & Privacy, under the General tab. Click Open Anyway to confirm your intent to open or install the app.
The warning prompt reappears, and you can click Open.

https://support.apple.com/en-us/HT202491

Please see that answer above.

Please see that answer above.


Load Time

The load time for all of the images for the website is awful. Took at like 3 minutes to load all the carousel images. The Meet the team pictures took like 1 min to load. Then you have to hover over all of them to load their black and white version. Perhaps add a loading screen overlay as the images load?

@cedric2018 the colored wont need an overlay right? because the skyline and the header, allow for the team pictures to load. Any way to load the black and white pictures before?

@mahirk Colored still takes a while. People won't stay on the skyline for very long. Regarding black and white loading here's a strategy that I learned from work: what you do is have it load ONE png where it's the colored version and the BW version right next to each other. But the div that the background image is contained in is only big enough to show that one photo. Then when you hover over the image, you simply change the background-position attribute to the position of the BW photo.

@macklees I really need on this one. Can you reduce the images size?


Define declarative test framework models and interfaces

Description of changes:
Creates the schema for the scenario files and defines a loading mechanism for the accompanying Python helper classes
The Python helper classes (subclasses of the FrameworkHelper abstract base class), provide Python implementations of the assertion and data fetching methods (using boto3). These classes are registered to the test cases using a @register_helper decorator.
Scenarios are defined using pydantic models - which allows for easy validation and encoding/decoding from YAML. The ScenarioRunner provides the main logic for iterating over steps and taking each action as defined by the step type. I have provided an example implementation for the create type step.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

/reopen

/reopen
/lifecycle frozen

/reopen

/lifecycle frozen

/test all


Set ack deadline of push subscriptions to max function runtime.

This will be superseded by direct schedules but will sort out messaging timeouts in the interim

Codecov Report

Merging #280 (9d0428a) into develop (73948de) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff            @@
##           develop     #280   +/-   ##
========================================
  Coverage    33.09%   33.09%
========================================
  Files           49       49
  Lines         4804     4804
========================================
  Hits          1590     1590
  Misses        3063     3063
  Partials       151      151

Flag
Coverage Δ

unittests
33.09% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/provider/pulumi/gcp/cloudrunner.go
78.76% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 73948de...9d0428a. Read the comment docs.


Update layout

Better highlight of sync mode

see comment answer with pic

see comment answer with pic
best regards


Backup

Think it would be useful to add a method to backup all DBs. What do you guys think?

That would be a great idea

Is this issue still active?
A method to backup a single DB or all DBs would be quit handy. I'm thinking about something similar to influxd backup

Is there any update regarding this matter?

It would be really nice to have this feature

Is there any update?

No updates here to adding this into the python client. I'm not sure this is actually a feature that we should address here, as the influxd binary gives you this now.


[SPARK-46911][SS] Adding deleteIfExists operator to StatefulProcessorHandleImpl

What changes were proposed in this pull request?

Adding the deleteIfExists method to the StatefulProcessorHandle in order to remove state variables from the State Store. Implemented only for RocksDBStateStoreProvider, as we do not currently support multiple column families for HDFS.
Why are the changes needed?

This functionality is needed to so users can remove state from the state store from the StatefulProcessorHandleImpl
Does this PR introduce any user-facing change?

Yes - this functionality (removing column families) was previously not supported from our RocksDB client.
How was this patch tested?

Added a unit test that creates two streams with the same checkpoint directory. The second stream removes state that was created in the first stream upon initialization. We ensure that the state from the previous stream isn't kept.
Was this patch authored or co-authored using generative AI tooling?

cc @HeartSaVioR

cc @HeartSaVioR, addressed all comments

GA failure is only from docker integration test which is unrelated.

Thanks! Merging to master.


June 2023 Release Issues

Hello!
SDK was updated to the latest version (June 23 release) and a number of issues were encountered:

No File -> New -> Maxim Microcontrollers. in Eclipse anymore.

New build errors for previously building projects. It looks like some Source/Include folders were removed in Libraries/Boards/MAX_XXXXX that contained high level board code such as pb.c/.h for pushbuttons and led.c/.h for LEDs.

Code builds fine once references to these files is removed. Workaround is just to reimplement these functions.
Thanks

Following these steps I was able to rectify the first problem.

@ConWalsh you didn't miss it, I added it to the release notes thanks to your findings.  So thanks again for your report!

We just pushed a hotfix for the Eclipse plugin as well.
As of 1.0.4-31 the new project wizard should be fixed as well and visible in the usual spot.


Adding Write::write_fmt counterpart in futures::io::AsyncWriteExt?

AsyncWriteExt is missing write_fmt counterpart from Write::write_fmt. I searched issues and PRs and it seems there is no one talking about it.
I tried to implement a write_fmt in my private futures-util build and it's seems pretty straight forward. async-std implemented their own version of write_fmt as well.
I'm wondering if there is any reason why not implement write_fmt in AsyncWriteExt?

AFAIK it's impossible to provide anything better than writer.write_all(format!(...)).await?; with the current std::fmt::Arguments design (and last I looked async-std's implementation was strictly worse than this because it missed out on using the expected length when allocating the String).

If this is always worse than write_all(format!(...)), I don't want to add this. Also, I don't want to add something like our own fmt::Arguments.


[SPARK-42859][PS][TESTS][FOLLOW-UPS] Delete unused file test_parity_template.py

What changes were proposed in this pull request?
Delete unused file test_parity_template.py
Why are the changes needed?
it is not used in CI (not listed in modules.py)
Does this PR introduce any user-facing change?
No, test-only
How was this patch tested?
CI

thank you @itholic , merged to master


Cannot find reusable code

Hi!
I was wondering where the bulk of your logic is in this project. You mention in the ReadMe that the code is migrating, what does that mean?

Hi most of the code was stored in separate repository. It was moved in here yesterday. I'm still working on documenting it, but you can find it here.


modify primary key

I believe the migration should be included here as well: https://github.com/Netflix/metaflow-service/blob/master/services/migration_service/api/init.py#L6

I believe the migration should be included here as well: https://github.com/Netflix/metaflow-service/blob/master/services/migration_service/api/init.py#L6

yep, diff isn't ready to be merged in yet. Still testing to determine if we should make this a major or minor version upgrade (although the change doesn't break the api, the migration may cause some downtime on the db since we are touching the primary key.


Fix typo

Changed all occurences of isMoblie to isMobile.
There were no open pull requests so I also rebuilt the source and example with webpack.
If you want a PR with only source files changed, I can resubmit.

thank.


HSEARCH-2048 Generalizing MetadataProvidingFieldBridge in order to al…

…so expose the type of created fields

Btw. the consumption of the added meta-data is limited to the sorting aspect for now. Obviously, the type info (and any other meta-data we may add down the road) needs to be consumed as well later on.

@Sanne Pushed an update with some doc clarifications/fixes.

merged in both master and 5.5.
I've added one more commit to make the example implementation in the docs fit in the rendered area w/o needing horizontal scrollbars.


Create saksham.html

Good work. I am merging your PR.
Do give us a star by hitting the star button of this repo!


Emit a dependency file that makefiles can include.

At the end of a build we output a dependency file in the object
directory containing a dependency list in make's format. We make no
effort to sort out the details of the dependencies: we just collect
all the file deps from the latex and bibtex tasks and make them
dependencies of the output pdf; the important thing is just that any
changes get detected so latex run is invoked, which will then figure
out the details.
If the output file is paper.pdf, we default to dumping the deps file
as latex.out/paper.pdf.d.

Interesting. Do you find this more useful than the phony rules in practice?
Unfortunately, these generated make dependencies are not simply coarser-grained, but less correct than latexrun's internally managed dependencies. That is, with this change, not only are there cases where make will unnecessarily run latexrun (which is probably fine, since latexrun will just exit immediately), but there are cases where make will fail to run latexrun when it is necessary.

I find it really annoying when make does things when there is nothing to be done, even if that is just shelling out to another build system. I'm not sure if there is a more practical complaint than that, though.
Interesting. Do you have an example of when this happens?

Interesting. Do you have an example of when this happens?

The most obvious one is that latexrun understands when environment variables that matter to TeX change. This includes things like input paths, format paths, and font paths. There are subtler things, too. For example, if the job fails or is interrupted after the first iteration has committed, but before it has stabilized, make will think it's done because the output file is newer than all of the inputs, but latexrun knows that the inputs have changed even though they're older than the output.

Submitted a better approach.


[FAQ] Error codes in config protocol.

[ ] I have searched the issues of this repository and believe that this is not a duplicate.
[ ] I have searched the release notes of this repository and believe that this is not a duplicate.

Describe the feature

The progress is 5%

The progress is 5%


Disable ci-link on new UI

Fixes https://github.com/refined-github/refined-github/issues/6453

The feature currently only works on the repo root, which is exactly where it isn't needed. See screenshot.
Test URLs
https://github.com/refined-github/refined-github
Screenshot

Old UI not affected


Fixed add/change arguments in FileAdmin.render_change_form

FileAdmin.render_change_form forces add / change argument values, while it's feasible to use add view to create files without using the clipboard

@stefanfoulis could you review this?

@mkoistinen any more fixes needed to merge this?

@czpython this way should be actually better

👍

LGTM, are there any tests? Also, please update the changelog.

@mkoistinen changelog added, test added too.


A few small commits

Each of the commits are hopefully self explanatory:

small typo in setup
Typehints, typechecking and removal of mutable defaults (as previously discussed with @tsrobinson)
Basic code cleanup using Black
removal of antipattern (== bool)
addition of cont_kdes param to docstring (please add a description or remove.
decorated staticmethods
added a doctring for the class (formatting)
removal of antipattern (== bool) and removal of additional non-required \
add simple sanity checking for savepath

Thanks @David-Woroniuk -- I will review and merge. This all looks great and thanks for carefully documenting everything!!

@tsrobinson - no problem. I didn't go for a deep dive on the code, but have a few additional ideas which we can discuss offline :)


Add ability to disable integrity check

Integrity check is really important but for development purposes it can be handy to disable it. Then, developer will be able to alter the data by hand in his editor of choice.

Maybe instead of introducing a new Store option it will be better to skip integrity check for specific file when it's *.sum file is also altered with some special word like "ALTERED"


🐛 Fix e2e waiting for status

What this PR does / why we need it:
This fixes the flake we have been seeing in the e2e tests where it gets stuck scaling up the KCP in remediation.
In the e2e tests we often have to wait for BMHs/M3Ms/Machines to be in some specific state. In some places we had mistakenly not added any conditions to the Eventually call, or used it in a not supported way. This commit makes the use of Eventually more consistent and fixes the missing conditions.
I'm also working on a larger follow up PR for factoring out the common code and turn it into functions.

/test-centos-e2e-parallel-main

/test-centos-e2e-main

/test-centos-e2e-parallel-main
/test-centos-e2e-main

/test-centos-e2e-parallel-main

/test-centos-e2e-main

/lgtm
Thanks for finding these was really tricky to catch when they can pass the test

/test-ubuntu-e2e-parallel-main

/test-centos-e2e-parallel-main
/test-centos-e2e-main
/test-ubuntu-integration-main
/test-ubuntu-e2e-parallel-main

/test-ubuntu-integration-main

/cherry-pick release-1-1
/cc @mboukhalfa thanks for reminder.

/cherry-pick release-1.1


kafka.errors.CommitFailedError: / max_poll_records

I have a vanilla Kafka setup in Docker, and a really basic worker connected to it.  The problem I find is after 6-8 message are processed, this message pops up in the logs:

[INFO] Job 5874a4627987449e85d6e974ac4b7a7f returned: None
Traceback (most recent call last):
  File "worker.py", line 39, in <module>
    worker.start()
  File "/opt/conda/envs/bidinterpreter/lib/python3.8/site-packages/kq/worker.py", line 260, in start
    self._consumer.commit()
  File "/opt/conda/envs/bidinterpreter/lib/python3.8/site-packages/kafka/consumer/group.py", line 526, in commit
    self._coordinator.commit_offsets_sync(offsets)
  File "/opt/conda/envs/bidinterpreter/lib/python3.8/site-packages/kafka/coordinator/consumer.py", line 518, in commit_offsets_sync
    raise future.exception # pylint: disable-msg=raising-bad-type
kafka.errors.CommitFailedError: CommitFailedError: Commit cannot be completed since the group has already
            rebalanced and assigned the partitions to another member.
            This means that the time between subsequent calls to poll()
            was longer than the configured max_poll_interval_ms, which
            typically implies that the poll loop is spending too much
            time message processing. You can address this either by
            increasing the rebalance timeout with max_poll_interval_ms,
            or by reducing the maximum size of batches returned in poll()
            with max_poll_records.

I've played with max_poll_interval_ms and max_poll_records but it seems like no matter how I set these, the problem always comes back.  Any help appreciated!

If anyone else is searching for this from the web, I found this particularly helpful:
https://stackoverflow.com/questions/61207633/how-to-process-a-kafka-message-for-a-long-time-4-60-mins-without-auto-commit
The average amount of time it takes for a message to be processed (document processing), is about 15 seconds.  I can process about 5 messages before it throws this error.  After increasing max_poll_interval_ms to 10 minutes, I still have this problem.  I'll post any updates but perhaps this is more to do with the consumer than the worker?

If anyone else is searching for this from the web, I found this particularly helpful:
https://stackoverflow.com/questions/61207633/how-to-process-a-kafka-message-for-a-long-time-4-60-mins-without-auto-commit
The average amount of time it takes for a message to be processed (document processing), is about 15 seconds.  I can process about 5 messages before it throws this error.  After increasing max_poll_interval_ms to 10 minutes, I still have this problem.  I'll post any updates but perhaps this is more to do with the consumer than the worker?

Hi @dyerrington,
kq is a thin wrapper on top of kafka-python. You might have better luck bringing this up with them. Thanks.

Hi @dyerrington,
kq is a thin wrapper on top of kafka-python. You might have better luck bringing this up with them. Thanks.

Hi @dyerrington,
Just checking if this is still an issue. Any luck?

Thank you for pinging for follow-up @joowani.  After a weeks-long investigation, I've determined that there is something specific to the Dockerized version of Kafka that I'm using that is different from the version in my development environment which I installed manually.  After some time, I learned that setting max_poll_interval_ms to 2x my maximum document processing time on the consumer side of things actually was the right place to use this.
Originally I was looking at this setting from the server-side and not the worker/consumer side of the problem.  As of yesterday, I haven't had these problems turn up since making this adjustment in my production systems.
For the record, and if it helps anyone else in the future, I'm using @wurstmeister's compose setup in production and it works great:  https://github.com/wurstmeister/kafka-docker

Awesome. Glad you got it resolved! Closing.


ports: Build NimBLE with 32-bit environment

Some code in NimBLE assumes that pointers are 4 bytes. Until this
is properly fixed build with -m32 to make sure resulting binary
is no misbehaving.

@gavin-yy  FYI

Is there same 8-bit/16-bit/64-bit board supported by (Apache) NuttX that we could use to validate NimBLE on environments different from 32-bit ?

@sjanc arm-none-eabi-gcc doesn't support -m32:
arm-none-eabi-gcc: error: unrecognized command-line option '-m32'; did you mean '-mbe32'?
make[3]: *** [/github/workspace/sources/apps/Application.mk:157: /github/workspace/sources/apps/wireless/bluetooth/nimble/mynewt-nimble/porting/nimble/src/os_mbuf.c.github.workspace.sources.apps.wireless.bluetooth.nimble.o] Error 1
arm-none-eabi-gcc: error: unrecognized command-line option '-m32'; did you mean '-mbe32'?

It's good to enable it in the common place?


Open the calendar to the current month

Hi, is it possible to start the calendar from the current month?
I mean, I have created a start date
let startDate = formatter.date(from: "01/01/2018")!
but I want that the calendar automatically scroll down to the current month (for example now we're in March so I'd like that the the calendar automatically goes to march).
Thanks!

in viewDiDLoad()
calendarView.scrollToDate( theDate )
calendarView.scrollToDate( theDate, animateScroll: false )

Thanks, I did not expected it  to be so easy!

cool.


Delete incoming relationships when an entity is deleted

When an entity is deleted, the authorization relationships nodes (e.g. the rCanAdmin relationship) from users and groups are not deleted.

Subject updated because it does not concern only authorization relationships, but all incoming relationships.
Which is good news as it can be done in a generic way when an entity is deleted.


Be more explicit about what's state in dismissable

Move _activeCardDragEndPoint near build() so it's more obvious that it
is part of the build state.
Make a couple of functions use setState() since they modify variables
that are used by build().
Add a more detailed comment to the empty setState() call, since those
are dubious in general and need explaining when they occur, lest people
start using them as magic incantations to Make Things Work.

LGTM


Consistent view emits using indexer's GRVs and committed versionstamps

View indexer saves both the GRV it used during the view update, and
the committed versionstamp in the couch_jobs job data section. Then,
the view reader uses those versionstamps to emit a consistent snapshot
of the view.

The committed versionstamp ensures that the view results can be
emitted even if the view gets updated between the time the view
finished and the reader gets notified.

The indexer GRV ensures that the view will emit the same doc
revisions in case when include_docs=true option is used as what it
read during the time it indexed the data.

The view reader uses those two versionstamps only if it initiates the
view build itself and then waits for it to build, to ensure that it
doesn't operate on stale GRVs.
Because the committed version is only available after the main
transaction commits, during view indexing finalization there is now a
separate transaction which runs at the end which reads the committed
version then marks the view as finished.
Since included docs have to be read at the indexer's GRV version, and,
that version is different than the committed version, those documents
are loaded in from a separate process.
There are a few complications introduced by this commit:

The versionstamps, especially the indexer GRV ones, may become
stale (older than 5 seconds) quickly and start throwing 1007
(transaction_too_old) errors. This could be mitigated by forcing
the indexer to commit after a shorter interval (1-2 seconds).

There is some fragility introduced in respect to how included
docs are loaded in a separate process. When that crashes or does
timeout it maybe throw a new type of un-expected error that we
don't catch properly.

For this to properly work it needs at least two more updates:
https://github.com/apache/couchdb/commit/2e40063aae17658138bfaffa2304b5e461e43e30
https://github.com/apache/couchdb/commit/dbbab649dc5cbca50664ab3f7c33147e6010a106
Issue: https://github.com/apache/couchdb/issues/3381

Thanks for taking a look @kocolosk. I integrated c0b24da to make the test suite pass. Should I integrate 2e40063 as well? I'd be happy to do it

Another piece that would be related is to modify the couch_views batcher so it doesn't try to bump against the 5 second timeout limit and instead use a lower limit (you suggested 1 sec). I think it would be a separate PR though

Agree that would be a separate PR. I'm fine to leave 2e40063 out as well -- it's not related to the work of getting include_docs to match the view output, it's on a different topic of ensuring the view actually observes a snapshot of the DB as it existed at some point in time (which is only the case today if the indexer does not need to retry, or the write pattern is insert-only).

Found one more test that was failing because of how it mocked the indexer and expected do_update to be called. Fixed in a separate commit 771bd2c4d69c950db8bc2f807bbda3f84f3b4a0c


How can I use Y.Map when I init the store?

Thx for this good repo!
I found that I can't use Y.Map when I init the store. Am I doing wrong?
export declare type MappedTypeDescription<T extends DocTypeDescription> = {
    readonly [P in keyof T]: T[P] extends "xml" ? Y.XmlFragment : T[P] extends "text" ? Y.Text : T[P] extends Array<any> ? T[P] : T[P] extends object ? Partial<T[P]> : never;
};

the object cannot be auto parsed as Y.Map

I got it. Just use the obj[key] will be fine. If doc notices that will be great!


[Feature] Add New-VSTeamPullRequest command

Currently there is no New-VSTeamPullRequest function
This would be a great addition and would enable more complex automated end to end, branch to branch feature development practices like
PR into Dev Branch -> build completes -> PR is merged -> Merged Branch Builds -> Release runs -> New PR to Test Branch -> PR Completes -> Test Branch Builds -> Test Release Runs > New PR to Prod Branch -> PR Completes -> Prod Branch Builds -> Prod Release Runs
This allows a totally segregated & isolated model between Builds/Releases and the differing stages at which these run at although does add extra complexity

What would be the minimum set of parameters to support for a v1?  There are a lot of options PR REST API Supporting them all will take a long time and make the function hard to use.


update: refactored notation-action repo

This PR refactors the repo. The refactored Github Actions are tested by this build.
Resolves #6, #10, #11, #12, and #13

I would like to see if we can refactor out dist, package-lock.json, packae.json, and tsconfig.json.

Good idea. Will do some reorganize on those files.

As suggested by @shizhMSFT, reorganized the repo following pattern of codeql.
(Note: the node_modules folder is required now since it holds all the import dependencies of the project).

@Two-Hearts This PR contains 3 changes:

code refactoring
add Apache 2.0 license
support downloading using checksums

Could you split this PR into 3 PRs?

@shizhMSFT I removed license related file from this PR. Will create another one to add license later. However, for support checksums, this is one of the main purposes for the current PR. Because the logic is already in the sign action. I'm just pulling it out and relocating it in a common lib in this PR.


Elaboration

I'll open this as a collection thread for questions and discussions related to the elaboration phase.
I have started work on the elaboration implementation and will update this as I encounter new issues.
1. Duplicate type declarations
Are type declarations within a super type visible in the derived type? If yes, how do we handle duplicate type declarations?
Are they always forbidden? Do we perform a memberwise comparison to test if they are identical?
2. Cyclic type dependencies
My current implementation reports cycles in the inheritance hierarchy like this.
What do you think, should the marker be on the name before or after the extends keyword?

3. Constant evaluation procedure
My current idea for the evaluation of constants is the following five step algorithm:

Collect all declarators and group them by name
Collect all assignments and make sure they assign to one of the declarators
Calculate the assigned values
a) Evaluate the right-hand side of each assignment
b) Make sure all assignment to the same constant result in the same value
Merge declarations
a) Evaluate the widths of integer types
b) Make sure the types of all declarations of a constant are equal
c) (?) Make sure no two declarations for the same constant are in the same ISA scope
d) (?) Make sure no two assignments the same constant are in the same ISA scope
Validate types
a) Make sure each calculated value actually fits in the respective constant's type

The basic idea is to calculate the values first and care about types later, because the constant back and forth between determining types and values is what's currently causing us so many headaches.
My main question is how step 3a should deal with dependencies between constants.
Here is a simple example (A). y needs to be evaluated first, because x depends on y:
int x = y:
int y = 1:

Here is an example (B) of an impossible constellation. Neither constant can be evaluated because there is a cyclic dependency. I would propose to generate a similar error message to the one for cyclic type dependencies.
int x = y:
int y = x:

The most interesting example (C) is this: We again have a cyclic dependency, but a third assignment can break the cycle.
This would be resolvable with a fixpoint solver. Do we want that?
int x = y:
int y = x:
int y = 1:

Example C also has another issue: suppose the first two assignments were in an instruction set and the third in a core definition providing said instruction set. That would mean the instruction set produces an error, but the core compiles without issues. I'm curious about your input here.
Also, are 4c and 4d allowed? The spec isn't explicit on this.

Addendum to 3:
Another issue that came up: sizeof can complicate this further, since expressions could depend on the size of other constants, which in turn can depend on the value of other expressions, possibly creating circular dependencies:
unsigned<XLEN> XLEN = bitsizeof(XLEN);

I would propose to simply disallow sizeof and bitsizeof of constant definitions during constant evaluation. sizeof(int) and other types should be valid though.
4. Attribute for default values used during analysis
Julian and I were talking about how type validation of instruction sets should handle incomplete types like unsigned<XLEN>.
One idea was to introduce an attribute that can be applied to ISA level constants to specify a default value used during analysis.
Any suggestions what we should name this attribute?

Another issue that came up: sizeof can complicate this further, since expressions could depend on the size of other constants, which in turn can depend on the value of other expressions, possibly creating circular dependencies:
unsigned<XLEN> XLEN = bitsizeof(XLEN);
I would propose to simply disallow sizeof and bitsizeof of constant definitions during constant evaluation. sizeof(int) and other types should be valid though.

The example is illegal anyways as you reference a variable befor declaring it. I think disallowing sizeof and bitsizeof  is problematic since the following would not be possible:
InstructionSet A {
    architectural_state {
        unsigned XLEN;
        register<XLEN> R;
    }
}

InstructionSet B extends A {
    architectural_state {
        unsigned ACC_LEN=2*sizeof(R);
        register<ACC_LEN> ACC;
    }
}

Core C provides B {
    architectural_state {
        XLEN=20;
    }
}

Wrt. 4: To my understanding you cannot do type validation on InstructionSet because of the parameterization (as in the example above). The validation can only be done at the Core definition level. Therefore I use in our generator class a GenerationContext which holds the Core definition to derive the values of constants and sizes of type. This context the also allows to cache the  values so that the AST only needs to be traversed once for a particular value within a particular context

Oh, okay. The specification doesn't mention that expressions can only contain constants declared earlier.
Adding that as a rule would unfortunately not prevent cyclic dependencies, as you can see here:
int XLEN;
int<XLEN> reg;
XLEN = bitsizeof(reg);

My current implementation is basically an iterative solver. It starts off knowing none of the values and in each iteration it calculates new constant values determinable only from the ones known so far. If none can be determined anymore, the remaining must have a circular dependency.
If we need sizeof, I can probably treat each constant's size as another implicit constant that can be depended on and depend on other constants.

2. Cyclic type dependencies
What do you think, should the marker be on the name before or after the extends keyword?

After the extends makes more sense to me, because the extends-clause is what constitutes the error.

3. Constant evaluation procedure

The procedure you propose sounds right to my. Yes, you definitely will have to iterate the evaluation until a fix point is reached (or detect cycles and error out). My mental model for the implementation parameters is that they behave like variables, syntactically. So they must be declared before use (either in the current architectural_state section, or in one of the parent ISAs); multiple declarations per ISA are prohibited; multiple assignments are ok. Assignments are evaluated in "program" order.
+1 on allow sizeof and friends; Eyck's example unsigned ACC_LEN=2*sizeof(R); is a good one I think.
Should we restrict the types that can be used on implementation parameters to unsigned<...literal...>/signed<...literal...>, to mitigate the back-and-forth between constant evaluator and type checker? We could also add a new alias param for a "big enough" integer type, as in param XLEN = 32;.

BTW,
unsigned<32> XLEN [[assume=32]];

means to me basically the same than
unsigned<32> XLEN =32;

since I can assign the value later on....

Let's keep it simple and simply forbid redeclaring user types.

What about core definitions? Multiple inheritance means a core can see multiple distinct types of the same name.

Yes, you definitely will have to iterate the evaluation until a fix point is reached

Assignments are evaluated in "program" order.

These two statements directly contradict each other. Having a fixed evaluation order is impossible.

Should we restrict the types that can be used on implementation parameters [...], to mitigate the back-and-forth between constant evaluator and type checker?

There is no point in doing that now since I already implemented the more powerful variant that allows arbitrary (compile-time evaluable) expressions (#49).

[W]riting an InstructionSet means flying blind, as in C++ templates.

I'm not sure I understand you right... are you saying we don't want any validation for instruction sets? C++ templates don't let you "fly blind", any decent IDE will report most errors you make in there, even without knowing the exact instantiation types.
When we last spoke about this, you said we should validate as much as possible without the full picture. The "default value" attributes would work like IntelliSense annotations in C++, providing example values with which the analysis would check program validity. Without default values, every expression that accesses unassigned state variables would be impossible to validate.

As a language user I would expect some influence in the implementation if writing some attribute. Therefore to me this would be missleading.

Using attributes and annotations only for analysis is common practice across pretty much all languages I have worked with. For example [[nodiscard]] in C++, [DebuggerStepThrough] in C#, @SuppressWarnings() in Java, etc.

unsigned<32> XLEN [[assume=32]];

means to me basically the same than
unsigned<32> XLEN =32;

since I can assign the value later on....

That is not how constant assignments work. If you assign 32 when declaring the constant, it can never be re-assigned to a different value (only re-assignments to 32 are valid), effectively making the parametrization useless.

In the statement
unsigned<32> XLEN =32;

is XLEN not a constant, it is just an unsigned variable. But the value XLEN is fixed in the instructions section. At least this is as we (@jopperm and me) discussed and defined it in the language. Constants in the architectural_state section are declared using const .

For each parameter/reset value in the combined architectural state of a Core, a CoreDSL 2 frontend must be able to evaluate all assignments to a unique value at compile time. In other words, there must either be a single assignment in the combined architectural state, or in case multiple assignments exist, the expressions on their right-hand sides must evaluate to the same constant value after a fix point is reached.

Why do I even bother implementing what the specification says when the specification is apparently completely wrong?
Those are 10 hours of my life I'm not going to get back.

I'll setup an offline meeting to discuss this.

We decided to change the parameter elaboration to make the last assignment effective.
@eyck, @AtomCrafty: Please review my changes to the spec. I also added a paragraph to the rationale page to document our motivation for this change.

I would change the wording Cyclic dependencies between parameters must be detected and reported as errors. (elaboration.md) to simply Cyclic dependencies are not allowed. The first statement refers to implementation details....

It seems that the changes reflect what we discussed, however the ArchStateItem grammar on this page doesn't make any sense. It allows several nonsensical statements:

a;
const x = 4;
x = 3 [[attribute]];

The first is legal althoug results in a NOP. The last one might make sense if the assignment shall be qualified.
The second one is definitly garbage. So I guess it should read
ArchStateItem ::= ('const' TypeSpecifier)? ID '=' ConstantExpression Attribute* ';'

[...] the ArchStateItem grammar on this page doesn't make any sense [...]

Ah thanks for spotting this, this has to be a relic from an early version. At some point, I introduced separate subsections for ASI declarations and assignments, but have forgotten to update the pseudo-grammar.
Fix: I made the TypeSpecifier parts in the declarations-subsection non-optional.

@eyck The first is not legal, though admittedly it's syntactically valid. A program containing this statement should be rejected by the validator because a; is not a valid expression statement.

I suppose you are both right: a; should be a legal statement in an instruction's behavior, but this form is not on the list of sanctioned statements for the architectural_state section.

I agree, this should yield a warning as most C/C++ compilers do.

Issue: because of the way the architectural_state section is parsed, we can't know whether a declaration or an assignment comes first in the code. So the following two snippets result in the same AST, even though one is invalid and the other isn't. I do not want to manually retrieve the corresponding CST node to compare line numbers, so I guess we need to change the grammar again.
architectural_state {
   int x;
   x = 5;
}

architectural_state {
   x = 5;
   int x;
}

I do not understand the problem. Both examples are a sequence of statements. From the point of assignment only statments before the assigment are considered. Even if I collect all architectural states accoding to the traversal order I get a sequence where the same approach applies. So what's the issue here?
Actually this was the way the scoping was implemented (at least in the initial version). So in the latter case the Scping would not return any candidate (since x has not been declared before the assignment) so the framework would flag the assignment as having an error.

The issue is that with the current implementation it is impossible for the scope provider to know which declarations come before a given assignment, because they are parsed into two separate lists. It is not known to the scope provider how these two lists were interleaved in the original code. You are right that this used to be solved differently, but I changed how the section was parsed in order to simplify the grammar a while back, which wouldn't have been a problem had we kept the old specification. I now have to revert said grammar change because the order of statements is relevant with the new semantics.

Just for reference, you mean the separate lists here, correct?
https://github.com/Minres/CoreDSL/blob/30f384a0a7741002dd45f6c04c25d43fa32b51ee/com.minres.coredsl/src/com/minres/coredsl/CoreDsl.xtext#L18
I agree, seems that a rollback is necessary here. Maybe you can offer getDeclarations() and getAssignments() helpers somewhere?

Closing this, at it has been addressed recently in #59.


Add strings from skos:altLabel & skos:example to FlexSearch index

Requested by V.P. via email on 2021-02-21:

Die Suchfunktion ist ein tolles Feature der
Visualisierung, die Suche sucht aber nur in den Klassenbezeichnungen und
nicht in den Scope Notes und Examples. Ist dies mit Absicht so gewählt
oder einfach schwieriger zu implementieren?

It looks like that currently the FlexSearch index is only built from skos:prefLabel in all languages that are provided (although there seem to be some problems with this, see #127). The use case to find concepts via altLabel or example definitely makes sense to me.

... The use case to find concepts via altLabel or example definitely makes sense to me.
Oh, yes, it would be good to have this feature. Then the content of altLabel should also be visible on the concept's page.

Then the content of altLabel should also be visible on the concept's page.

Thanks for the request. I opened an issue at #142 .

Also notation is not indexed. See #161

With using the "full" tokenizer with https://github.com/skohub-io/skohub-vocabs/pull/180 adding more fields could blow up index size for bigger vocabs with lots of alt labels and maybe long scopeNote texts. We will have to take this into account when addressing this.

This could also be configured with the config file #185

#267 also suggested to search for definitions
To summarize, these fields are wanted to be searched additional to skos:prefLabel:

skos:altLabel
skos:example
skos:notation
skos:definition

This might blow up the index clients have to download. @acka47 maybe we are now reaching the point, where we need support for a dedicated search engine.
Tasks:

[ ] make search fields configurable
[ ] check size of created index
[ ] (maybe add a dedicated search engine)

maybe we are now reaching the point, where we need support for a dedicated search engine

Yes. I could imagine the following approach:

using FlexSearch for smaller vocabs, e.g. value lists (something like up to 50 concepts)
for larger vocabs, implement lookup via the reconciliation API suggest endpoint (reconciliation support only makes sense after a certain amount of concepts anyway)

To work on this issue I decided to first fix #186.
This was long due and will help us on this issue.
Question I have is: How do we want to show the user that a concept matches though it is not the skos:prefLabel that matches?
I'm thinking of displaying the concept in italics and then maybe providing something like a tooltip?
Any other suggestions? Will post something here when I have a first prototype

What about somehow adding the field in brackets after the preflabel, like:
Search for "dialekt" in https://test.skohub.io/hbz/lobid-vocabs/heads/master/nwbib.de/subjects.de.html:
Show:

740000 Sprache

744000 Mundarten (Alt Label: Dialekt)

I think I'm pretty close. I just asked myself why just:

skos:altLabel
skos:example
skos:notation
skos:definition

shouldn't hiddenLabel, scopeNote and note be searchable as well?

tagging also @carlschuurbiers if he has any opinions on that

sure, that would be very useful, I think if one is searching for some expression, you will probably want to look everywhere. I only hope it will not clutter the interface too much if you add a checkbox for every expression... But it is a great feature!

is it possible to highlight the other places where an expression was found in the right window? So if nothing is highlighted in the left window, one can click on the entry and in the right window, where all data is displayed, the corresponding text is highlighted?
adding just the altLabel in the left window is great though. But we can't add all labels there...


Documentation for private/hidden tasks is ambiguous

Without experimentation, it's not clear if private hidden tasks with a name that starts with underscore mean that you should do _task_private_task() or task__private_task() or something else entirely.

In case anyone is curious, you can either use task__private_task() or in the return dictionary, you can have a basename key, 'basename': '_private_task'

well, it is no ambiguous to me. The name of task is what comes after task_.
Anyway it would be nice to have an example in the docs...
Would you please submit a pull-request to the docs.


Allowing dependabot to update GH Actions

The @pyansys/pyansys-core team has recently discovered that dependabot can automatically update GH actions (in a similar fashion to requirements). Proposing implementation in various repositories to keep them up to date.

@RobPasMue So, you mean something like this
- uses: actions/checkout@v3

would be proposed to upgrade to v4?

Precisely! 😄

See an example right here: https://github.com/pyansys/pydpf-post/pull/129
PyDPF-Post already had it implemented.

I'd need another reviewer for this PR (when possible 😃 )


smbmap: unstable-2023-03-29 -> 1.9.1

Changelog: https://github.com/ShawnDEvans/smbmap/releases/tag/v1.9.1
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 237184 run on x86_64-darwin 1

2 packages built:

smbmap
smbmap.dist


Add pip install cryptography package for python image

Signed-off-by: Veena Saini<EMAIL_ADDRESS>
Description of the changes
Recently gsc sign image started throwing error "No module named 'cryptography'" for python base image. Issue Link:
https://github.com/gramineproject/gramine/issues/482
This PR resolves the above issue by adding pip install cryptography to Dockerfile.build.template files.
How to test this PR ?
Step 1: docker pull python
Step 2: ./gsc build --insecure-args python test/generic.manifest
Step 3:  ./gsc sign-image python enclave-key.pem    // this should pass
example link: https://gramine.readthedocs.io/projects/gsc/en/latest/#example

This change is

This is a wrong way to fix this issue. I already created two pending PRs to fix it:

https://github.com/gramineproject/gramine/pull/429
https://github.com/gramineproject/gsc/pull/49

My PRs are under review and will be merged soon. You can read through the discussions to get a rationale behind my proposed fix. I'm closing this PR as a duplicate.