Should we add a way to pass in external executable schemas?

Porting the discussion on external executable schemas to its own issue so we don't lose track of it. Original discussion is copy-pasted below:
@mfix22 https://github.com/gramps-graphql/gramps/issues/26#issuecomment-352119917

@jlengstorf I think this is a very neat way to merge data sources. Unless I am reading this wrong, the only use case we have that this does not cover is for linking in a remote executable schema:
makeRemoteExecutableSchema({
  schema: builtSchema,
  link: HttpLink, // which points at my remote schema
});

Technically we could stitch this remote schema in after gramps() has been called but that seems messy.

@jlengstorf https://github.com/gramps-graphql/gramps/issues/26#issuecomment-352122456

@mfix22 Yeah, that's a limitation of GrAMPS — in order to keep the data sources simple, we need to run all data sources through GrAMPS to get an executable schema. I suppose it would be possible to add an extra option to allow additional executable schemas to be passed in.
Right now, we create an executable schema right before returning.
We could modify this so you'd call something like:
const external = makeRemoteExecutableSchema({
  schema: builtSchema,
  link: HttpLink, // which points at my remote schema
});

const GraphQLOptions = gramps({
  dataSources: [XKCD],
  externalSchemas: [external],
});

And then we'd modify the gramps() function to spread that new prop as well:
  const schema = mergeSchemas({
    schemas: [...schemas, ...externalSchemas, ...linkTypeDefs],
    resolvers,
  });

Theoretically this would Just Work™ and would even allow schema stitching between GrAMPS data sources and remote executable schemas.
However, I haven't worked with remote schemas yet, so I'm not sure if this would do what I think it'll do. Let me know if this looks like it'll work and I'll push a new beta package so you can try it out.

@mfix22 https://github.com/gramps-graphql/gramps/issues/26#issuecomment-352225241

@jlengstorf This is exactly how I am currently handling the use case! I am pretty sure it will Just Work™ 😄 I think that option would fill in the gap perfectly while keeping data sources simple.

@kbrandwijk https://github.com/gramps-graphql/gramps/issues/26#issuecomment-352256297

@mfix22 @jlengstorf An alternative approach would be to keep the gramps part contained to a single schema, and leave the stitching of other schemas to other emerging tooling (separation of concerns). I don't think this is messy at all.

@jlengstorf https://github.com/gramps-graphql/gramps/issues/26#issuecomment-352257476

@kbrandwijk I agree, but I wonder if it's worth including this for simple use cases since it's really just two lines of code. I may end up eating these words later, but I can't really see how this could cause problems later on.
I do agree that advanced use cases should probably be handled outside of GrAMPS, but I see a good use case in something like this:

I have a gateway that's built of my own custom data sources
I'm stitching them together with GrAMPS
I have a new requirement that means I need to include GitHub repos for users
I add the GitHub schema as a remote schema and stitch it together with my User type via GrAMPS

If it went too far beyond that, other tools would be easier/recommended, but I can see this being valuable.
Any objections to including it?

@kbrandwijk https://github.com/gramps-graphql/gramps/issues/26#issuecomment-352258460

@jlengstorf Well, I guess it wouldn't hurt anyone, but I would strongly advise against making this a best practice. Mainly because in most use cases, it will involve more than just calling makeRemoteExecutableSchema. For most endpoints, there's authentication, there's .graphqlconfig.yml for defining your endpoints, there's not wanting the entire remote GraphQL schema to become part of your final schema, etc.

@jlengstorf https://github.com/gramps-graphql/gramps/issues/26#issuecomment-352258843

Hmmm... that's a pretty solid argument for not including it. I haven't worked on remote schemas, so I'm pretty much going with the experience of the people using it.
@mfix22, do you have a counterargument here?
If @kbrandwijk's point holds true for all but the most trivial use cases, it does seem smart to leave out the feature and instead add an example of how to do this outside of GrAMPS. Otherwise I'd worry we're inadvertently introducing something we'd have to either deprecate later or add a big warning recommending people don't use it. (And if either of those is true, we may as well support the proper path out of the gate.)
Thoughts?

@mfix22 https://github.com/gramps-graphql/gramps/issues/26#issuecomment-352622460

I do see @kbrandwijk's point about not wanting to expose the entire 3rd party's schema as a definite concern here. That was a question I still want to ask one of the Apollo guys: what is the best way to expose part of a schema, especially a remote one?
As for the other concerns, GrAMPs already supports adding context and you can still add headers to handle authorization use-cases (more Express middleware for example).
For initial release, i don't think GrAMPS needs to support remote executable schemas. The escape hatch to include them might help some users though. What is nice is that including them doesn't hurt the users that don't.
Maybe the best argument for not including them at first is that once they are included, it would be a breaking change to remove them, but that is not true the other way around (once we know more about exposing a subset of a schema).

@jlengstorf https://github.com/gramps-graphql/gramps/issues/26#issuecomment-352641566

@mfix22 The Graphcool team just released a pretty interesting article on exposing subsets of underlying GraphQL schemas. There's a video that shows an approach to controlling data access. Check it out: https://blog.graph.cool/graphql-databases-a-preview-into-the-future-of-graphcool-c1d4981383d9
Maybe we can pull in some Apollo folks to weigh in? @stubailo, @peggyrayzis, or @jbaxleyiii — any thoughts on best practices for managing remote schemas as laid out in @kbrandwijk and @mfix22's comments above? (Also, 👋.)

I would love to see this because I want to create a project using multiple microservices with micro-js.
are PRs welcome ?

@Goluis Absolutely PRs are welcome!
Could you provide some pseudo-code or a use case for the external schemas? Specifically, how are you putting this together that doesn't work with the typical GrAMPS approach of putting the data sources together into a single executable schema?
(I'm asking so I can understand how other devs are structuring their code, not to try and shoot you down.)

@jlengstorf Can you explain to me how the stitching part works exactly. I assume that is for stitching multiple datasources together? But I'm wondering why the stitching part is defined on a single datasource. Maybe I'm missing something. The reason I'm asking it here is because of its relationship to merging with other schemas (like the external schemas discussed here).

@jlengstorf of course!
Currently data-sources are supposed to be merged with a gateway as shown here and they can also run their own gateway, but in production is the app gateway the one who merges the schemas and deploys.
Using remote schemas two things change:

data-sources can be deployed too
the gateway merges a remote schema

Some benefits of this approach:

Parallelize workload by distributing ownership and responsibility across different teams.
Scale easily by adding more resources to independent microservices as needed.
Only data-sources need to be redeployed and the main gateway only needs to be changed when adding or removing data-sources

@kbrandwijk The idea behind stitching in GrAMPS is, in a nutshell:

I have two data sources that need to share data: User and Post
I set Post as a peer dependency of User (and vice versa)
The stitching prop of the User data source defines how the User type will expose data from the Post data source (and vice versa)

The idea behind this is to allow complex schemas to be broken up into data sources without forcing them to become silos.
I don't believe this would interfere with merging into other schemas, but we haven't tested that scenario yet.
@Goluis I think I see what you're going for, but I do want to clarify that the data sources are not independently deployable. The CLI will stand up a simple gateway for development, but the intention is that those data sources will ultimately be consumed by a GrAMPS gateway.
That being said, you could certainly put multiple GrAMPS gateways into production as independent µ-services and merge them via remote schemas into a main gateway.
One thing that GrAMPS is set up to do, though, is allow data sources to be included in multiple places.
So let's take those zeit docs as an example:
To start, we have the UI µ-service and a GraphQL endpoint:

ui.example.org
graphql.example.org

The GraphQL endpoint has three data sources:

Register
Post
User

As the product scales, you decide to split into two GraphQL µ-services:

graphql-register.example.org
graphql.example.org

Let’s assume both Post and Register rely on User for normal operation. Since GrAMPS data sources can be installed on multiple gateways, there's nothing to stop you from including the User data source on both µ-services:

graphql-register.example.org consumes the Register and User data sources
graphql.example.org consumes the Post and User data sources

These wouldn't need to be stitched together at all, and you should get the same end result.
Am I understanding your use case correctly, or is there something I'm missing?

the difference is that if the data-source for User is changed it will need to be updated in graphql.example.org and graphql-register.example.org, but if User were a microservice with a remote schema (graphql-user.example.org for example) and the other gateways are using a link to its schema then only graphql-user.example.org will need to be updated

Ah, good point.
I don't have any solid argument for not adding support for external schemas. And @kbrandwijk is thumbs-upping, which I assume means he's onboard.
@Goluis Does the API addition laid out in https://github.com/gramps-graphql/gramps/issues/47#issue-283307728 seem like it's going to work for your use case? If so, want to submit the PR? 😄

@jlengstorf Well, onboard with the reasoning from @Goluis that you don't want to include it in both, and having to update in multiple places. I'm still not 100% sure about the external schemas. But I think it comes down to the positioning of gramps(). Do you see it as a completely self-sufficient tool, then you need this as well. Do you see it as part of a toolchain, than I would focus on the main responsibilities of the package.

@kbrandwijk I guess I'd make the argument that the maintainers of GrAMPS can't really make that decision for the devs using it. 😕
I can see it working in both contexts, depending on the structure of a given team. Maybe you could help us put together some architectural considerations for using external schemas (e.g. all the things that make you uneasy, so other devs can be aware of potential pitfalls)? Maybe we could work with @schickling to put this on https://www.advancedgraphql.com?

I think we should do a boilerplate for GrAMPS, to be used by graphql create, so we can figure out how these tools fit together. Maybe we can work on that together?

Maybe we could work with @schickling to put this on https://www.advancedgraphql.com?
This would be awesome! @nikolasburk is planning to give Advanced GraphQL a full rehaul in January. Would be great if you'd like to get involved!

Here's the apollo example of how to add multiple remote schemas working: https://launchpad.graphql.com/q5kq9z15p
I already tried to add one of the example endpoints in top of gramps without success (I don't know why it didn't work, just that the schemas weren't merged)
// useRemoteSchemas.js
import { mergeSchemas } from 'graphql-tools'

export default (remoteSchemas, grampsOptions) => async req => {
  const options = grampsOptions(req)
  const remotes = remoteSchemas.map(schema => schema())
  const schema = mergeSchemas({
    // The remote schemas are fine but no merge occurs
    schemas: [options.schema, ...await Promise.all(remotes)]
  })

  return Object.assign(options, { schema })
}

// server.js
// Note: Only the relevant code is included
import { introspectSchema, makeRemoteExecutableSchema } from 'graphql-tools'
import { HttpLink } from 'apollo-link-http'
import fetch from 'node-fetch'
import XKCD from '@gramps/data-source-xkcd'
import useRemoteSchemas from './useRemoteSchemas'

const RemoteSchema = async () => {
  const link = new HttpLink({
    uri: 'https://v7l45qkw3.lp.gql.zone/graphql',
    fetch
  })

  return makeRemoteExecutableSchema({
    schema: await introspectSchema(link),
    link
  })
}

const GraphQLOptions = useRemoteSchemas(
  [RemoteSchema],
  gramps({
    dataSources: [XKCD]
  })
)

app.use('/graphql', graphqlExpress(GraphQLOptions))

Including remote schemas right now will require changes to the api and probably context, btw check out this current discussion about possible incoming changes to stitching schemas

Adding remote schemas will be a lot easier with the changes proposed in #62 and #63.


[Question] Why adonis is still using indicative 5 and not 7?

Issue doesn't adhere to the guidelines of creating a proper issue.

Make sure that you are only reporting bugs or submitting feature requests on Github.
In case of feature requests, please be descriptive and explain how this feature helps a broader audience and not just you.

If you are not submitting a bug and neither a feature request, then please use the forum or discord server
In any case, issues which are not descriptive enough will be closed. Please respect our time.


x/telemetry: present the list of approved counters in telemetry.go.dev

The list of approved counters is in the golang.org/x/telemetry/config module.
https://go.googlesource.com/telemetry/+/refs/heads/master/config/config.json
(this is linked from a section in https://telemetry.go.dev/privacy, but I don't think the privacy page is the first page users would think of when looking up such info).
Present it in a human-friendly way and make it more discoverable from the telemetry.go.dev main page.

Pragmatically blocked by #63504


fix(DASH): Fix MPD Patch when SegmentTemplate is shared between Representations

Fixes #7214

After talking to @tykus160 , it still can't be merged because there is an error in the compiled build.

Just curious if I can get a live version of this PR for test? So that I don't need to setup my HTTPS server.

@tykus160 Seems only <add/> works. I tested neither <remove/> or <replace/> works.
In practice, MPD:
<S d="1" t="0" />
<S d="2" t="1" />
<S d="1" t="3" />

Remove patch:
<remove sel="/MPD/Period[@id='1']/AdaptationSet[1]/SegmentTemplate/SegmentTimeline/S[t='3']">
</remove>

Replace patch:
 <replace sel="/MPD/Period[@id='1']/AdaptationSet[1]/SegmentTemplate/SegmentTimeline/S[t='3']/@r">
    3
</replace>

Error message:
Expected _class2({ startTime: 3, endTime: 4, trueEndTime: 4, ... }) to equal _class2({ startTime: 3, endTime: 6, trueEndTime: 6, ... }).

@tykus160 Oh yes you're right. Both <replace> and <remove> not works even I put <SegmentTemplate> into <Representation> and change @sel respectively. Should I open another issue for that?

Yes please!

@wilsonluniz-harmonicinc you can, it will make tracking of that easier.


Simplifiy access to version number before requesting 24h Forecast

It is possible to use "https://www.meteoswiss.admin.ch/product/output/versions.json" to get the last version of the forecast. It could be more stable and resilient as parsing the html code.

The hourly forecast is now included as part of the get_typed_data() call.   Look at the type ClientResult, it has a key forecast with a regionHourlyForecast key that contains a list of forecasts timed every hour for five days.

(I leave the old code for compatibility, but we should probably deprecate parsing HTML altogether.)


Certified Connectors - Incorrect Email

Office 365 is reporting that the listed email address<EMAIL_ADDRESS>is invalid and email cannot be delivered to this address.
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

 Apologies.


Fix issue #62

Summary
This is a rework of the code base to fix issue #62. This new version is even leaner than the original, which also improves the learning experience.
Changes
All files have been altered except the hint.txt. Here are the changes broken up by files:

code.py

The programmatic endpoint was removed since it had no real purpose and only went against the DRY principle.
The index function is restructured and doesn't leverage sanitization. This introduces the XSS vulnerability, as explained in the solution text.
The index function now correctly handles the case of a missing or empty planet parameter.

tests.py

It has been expanded with tests to check for a missing or empty planet parameter.
It now contains a test with a payload having script inside it.
The unused import of Flask was removed.
It demonstrates how to send a form parameter in a HTTP POST request inside a unit test.

hack.py

Deleted.

hack.txt

A new text file with instructions how to insert an active content into the application to trigger an XSS attack. The initial implementation will be susceptible to XSS attack, whilst passes with a fixed version according to the solution.

solution.txt

It is now fully correct and aligned with a program behavior.
It instructs the reader how to use the escape function in a non-deprecated fashion.

Closes:

#62

Task list

[x] For workflow changes, I have verified the Actions workflows function as expected.
[x] For content changes, I have reviewed the style guide.

Thanks! I want to also ask the original creator of the level who is @viralvaghela.

@jkcso I have reworked the game as we've agreed. It is now showcasing some advanced features of working with dynamic HTML pages (see the new details.html template page) as well as ways to circumvent proprietary sanitization approaches. The solution.txt explains all the details. The new tainted payload that triggers the XSS attack is &ltimg src='x' onerror='alert(1)'&gt, which nicely illustrates that attackers can always came up with creative payloads.
I hope that this version is our desired win-win situation.

Pushed manually to make some minor changes on the language (not code!) and resolve conflicts! Thanks a lot legend @evarga!


Configurable control hide delay

To synchronize the hiding of plyr with other UI elements, I'd like to configure the delay after which the UI hides but unfortunately, this delay is hardcoded to 2000/3000ms right now. Suggestion would be to introduce a new option hideControlsDelay to allow this delay to be configured.

Or if we can use different classes (there's actually a 4000ms delay too) with animation-delay. Not sure if this is possible, but I think this is better in the css.

but I think this is better in the css

While it's technically better to do in CSS, I think it might be a bit unintuitive for users having to modify a CSS class to adjust this delay.

I don't necessarily think adding custom css is hard, or that we should add things to the config API because it's easier unless it's a common request (which this hasn't been so far).
I don't have a strong opinion on the matter though. Your suggestion is reasonable and more straight forward to implement given what we have. Maybe PR it? I would try to find good names for all 3 delays in that case ex interactionDelays: {mouse: 2000, touch: 3000,  keyboard: 4000} (not sure this is exactly how they're being used).

I'd also like to have these settings as @friday mentioned.
As for now, I use these CSS solution to have youtube-like behavior:
@keyframes controls_animation {
	0%, 85% {
		opacity: 1;
		transform: translateY(0)
	}

	100% {
		opacity: 0;
		transform: translateY(100%)
	}
}

// Hide controls with 3 seconds delay
.plyr--video.plyr--hide-controls .plyr__controls {
	animation-duration: 3.5s;
	animation-name: controls_animation;
}


CB-8927 Add media to cspMeta and Windows as plat

Fixes CSP errors on media test.
Addes CSPs for Windows platform

LGTM.

Merged. Sorry, forgot to say "This closes #127." You may safely close and delete.

You need the nifty coho merge-pr script. :wink:

@muratsu Can you please close this PR?


[Config] Support env variables

Apparently Viper supports env variables unmarshaling after all, just not when the config
file does not contain the actual keys.
This PR updates the Dockerfile to use the default example config (which has all the keys)
as the file to put into /app/config, and updates the k8s deployment to use env variables
and secrets.
Closes: https://github.com/lloydmeta/tasques/issues/11

Codecov Report

Merging #15 into master will increase coverage by 0.04%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master     #15      +/-   ##
=========================================
+ Coverage   83.05%   83.1%   +0.04%
=========================================
  Files          18      18
  Lines        1151    1154       +3
=========================================
+ Hits          956     959       +3
  Misses        133     133
  Partials       62      62

Impacted Files
Coverage Δ

app/cmd/root.go
41.48% <100%> (+1.92%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 83b0cc4...9c7103e. Read the comment docs.


Verify presence of markdown folder before creating public/ folder

Fixes #11
Signed-off-by: Evgeniy Kosov<EMAIL_ADDRESS>

Great, thank you, @pippo !


etcdserver: check index of the kv when restarting

/cc @heyitsanthony @gyuho

lgtm

lgtm. thanks!

Hey @xiang90 would you mind looking at this if you find the time: https://github.com/coreos/etcd/pull/5780 -- This pull seems to have added the db corruption check, but that now leads to unrecoverable scenarios if the machine reboots, e.g. due to a power outage.

but that now leads to unrecoverable scenarios if the machine reboots, e.g. due to a power outage.

Can you give me more details? Have you met this issue in practice, or in theory?
Power outage should be fine. The index checking should be OK since we sync the db before sync the metatdata for matching.


Enable travis CI support

The travis.yml file is already there and working. https://travis-ci.org/ support has just to be enabled in the repository settings.

I guess the usefulness of this is contingent on whether or not I write any tests :p

Since their are some test it might be also useful to see whether the existing code breaks :D


Table names with commas.

Our SQL Server ERP software was installed with a table prefix that includes a comma. CI's Active Record automatically splits the FROM clause based on commas assuming they are separate tables ignoring quotes/brackets to keep the table name together.
i.e. "[Allen,LLC$Customer]" gets changed to "[Allen, LLC$Customer]" which fails.
p.s. The poor choice of table prefix is something we would like to change at the next opportunity, but yet this seems something the FROM function should handle better.

MySQL
Database and table names cannot contain “/”, “\”, “.”, or characters that are not permitted in file names.
http://dev.mysql.com/doc/refman/5.0/en/identifiers.html

@xmartinj
The closed issue #3764 gives an idea for a workaround. If you pass the selected fields as an array what would happen now?

If mysql accepts dots in schema names and fixes bugs related to it, why can't you? https://bugs.mysql.com/bug.php?id=61078 This IS a bug, as you're enforcing something that is not enforced by MySql. Dot names in schema name works fine everywhere else.

If MySQL can do it, why can't we? Really?
MySQL is a huge project with countless contributors, a powerful corporation behind it, and on top of that only has to deal with itself.
CodeIgniter is a 3MB framework that sparingly gets small contributions from outsiders, is largely maintained by me alone, in my spare time, and it touches on pretty much all elements of the web technology stack.
Even isolating the issue to databases alone - we try to maintain both compatibility and ease of use between more than 10 database platforms. And the issue isn't even closed yet.
That's why.

No worries. We've moved to Laravel, and are moving away from this database
as well.
On Mon, Mar 20, 2017, 7:01 AM Andrey Andreev<EMAIL_ADDRESS>wrote:

If MySQL can do it, why can't we? Really?
MySQL is a huge project with countless contributors, a powerful
corporation behind it, and on top of that only has to deal with itself.
CodeIgniter is a 3MB framework that sparingly gets small contributions
from outsiders, is largely maintained by me alone, in my spare time, and it
touches on pretty much all elements of the web technology stack.
Even isolating the issue to databases alone - we try to maintain both
compatibility and ease of use between more than 10 database platforms. And
the issue isn't even closed yet.
That's why.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bcit-ci/CodeIgniter/issues/2755#issuecomment-287729047,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABjWvVuGnolGkxBNmaC6L-zkda7jeregks5rnlybgaJpZM4BS1ur
.

After 4 years since this has been reported without any attention, anyone can presume this "feature" will never be implemented, even though, it is a bug. Your "holier-than-thou" attitude doesn't do much to convince me you listen to your user-base, so you can abstain from snarky comments. I'm sure you're full of them.

If MySQL can do it, why can't we? Really?
MySQL is a huge project with countless contributors, a powerful corporation behind it, and on top of that only has to deal with itself.
CodeIgniter is a 3MB framework that sparingly gets small contributions from outsiders, is largely maintained by me alone, in my spare time, and it touches on pretty much all elements of the web technology stack.
Even isolating the issue to databases alone - we try to maintain both compatibility and ease of use between more than 10 database platforms. And the issue isn't even closed yet.
That's why.

Andrey, are you maintaining primarily for your existing sites that use CI?
What public sites do you have that leverage CI?
On Mon, Mar 20, 2017 at 10:05 AM alex7071<EMAIL_ADDRESS>wrote:

After 4 years since this has been reported without any attention, anyone
can presume this "feature" will never be implemented, even though, it is a
bug. Your "holier-than-thou" attitude doesn't do much to convince me you
listen to your user-base, so you can abstain from snarky comments. I'm sure
you're full of them.
If MySQL can do it, why can't we? Really?
MySQL is a huge project with countless contributors, a powerful
corporation behind it, and on top of that only has to deal with itself.
CodeIgniter is a 3MB framework that sparingly gets small contributions
from outsiders, is largely maintained by me alone, in my spare time, and it
touches on pretty much all elements of the web technology stack.
Even isolating the issue to databases alone - we try to maintain both
compatibility and ease of use between more than 10 database platforms. And
the issue isn't even closed yet.
That's why.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bcit-ci/CodeIgniter/issues/2755#issuecomment-287768553,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABjWvU_n5OIhYOK1KHjbfeeA26zSxXzAks5rnoe0gaJpZM4BS1ur
.

@alex7071 I suggest you read, then read again your own reply, and repeat that excercise infinitely until you realize that my "holier-than-thou" attitude is a direct response to yours.
It's fine if you're unhappy with something, and I don't pretend that CI is perfect or even tries to satisfy everybody's wishes. Just don't act like an entitled brat and then talk about my attitude.
@xmartinj Not that I see how that is related to the issue, but no - I'm not maintaining it primarily for my own projects.
As for your second question - with all due respect, none of your business.

its a poor design choice

@narfbg uh-huh, i'll get on it and get back to you. Douche.

@alex7071 This is a public space, maybe you should never insult others...?

Why would one decide to include comma's or semicolon's in a table / database name? Isn't this bad practice?

@omphemetse How about dots? if i have the databases set up from ages ago with GB of data, like database.prod, database.dev when codeigniter had 0 problems with dots in names and i want to upgrade to a new version i have two options: NOT upgrade, or waste hours and hours of importing data in new databases, changing configs, re-testing the entire application, so i make sure some other dude did not do stupid things like hardcoding names, because of a bad decision that won't be corrected because some dude declares "it's not a bug" or "no time to fix, busy, lulz". I'm not wasting my time fixing it either because it won't ever be merged. Move on to better things seems to be the only sane choice and let this framework die in peace.

This issue remains open only as a reminder for us in case some day we want to, and have the time and resources to implement it.
But that being said, everything that could be said here has already been said. I'm locking the conversation. And since that Alex guy is clearly only interested in ranting, he's now one of the few members of our block lists.

Ok, let's officially end this ...
For the record, I don't like not having this feature. But attempting to implement it comes with too many potential problems to solve and it's just not happening. All tools come with some limitations and its time we all accepted this limitation.


Graphic options

Graphic settings such as running in windowed/fullscreen, and resolution.

This will be postponed to when a UI redesign is done to accomodate more settings

This should be added to the new "General" section, as it's mostly comprised of display options only (conventional graphic settings don't apply)


CloudIoTCoreMqtt::_backoff_at_millis initialized to very high number.

While working with this library I found that my ESP32 wasn't connecting to MQTT. I found that this was because CloudIoTCoreMqtt::_backoff_at_millis was set to a very high number, which prevented the code from passing the first if test in CloudIoTCoreMqtt::mqttConnect(). The value of  CloudIoTCoreMqtt::_backoff_at_millis wasn't consistent after resetting the micro-controller, but was always well above CloudIoTCoreMqtt::max_backoff.
I noticed that CloudIoTCoreMqtt::_backoff_at_millis was declared but not initialized in CloudIoTCoreMqtt.h; initializing CloudIoTCoreMqtt::_backoff_at_millis to CloudIoTCoreMqtt::max_backoff allowed the ESP32 to connect to MQTT (after 60 seconds, as expected). I'm assuming that initializing CloudIoTCoreMqtt::_backoff_at_millis to a lower value would still work, but I haven't put the time in to experiment with this.
Curios if anyone else has run into this issue, or found a more elegant solution.

perfect, thanx


Report Search Button Design

The "Create New Report" button is attached/overlapping with report box. Maybe also change "Create New Report" to "Create Report" to keep it shorter.

I know, I already messed around at that button. 'Stealing' CSS code from other buttons did not help to fix the design. I actually don't know why those other buttons have a different font and this one not.

Refer the PR #571 and please suggest modifications!

#571 fixes it!
Closing.


Remove support for invalid domain object names

Prior to this commit, the use of invalid characters in names of projects
and container objects was deprecated. Now, it fails the build by
throwing an InvalidUserDataException.
Resolves #6288.

I wonder if we should reconsider some of these. E.g. what really speaks against a space in a name? Sure, it's inconvenient to address it, but given the fact that so many people use spaces in directory names, it seems like they value readability over convenience. Also instead of failing we might want to escape those names instead.
Think about the use case where you call gradle init in a folder with a space. This will now fail. The only workaround is to create a settings file with a different name, but then init will no longer do anything. So this breaks Gradle init for folders with any of these forbidden characters.

Ok, how about we keep spaces deprecated but continue to support them and only fail on the rest of the invalid names?
How would you escape them? By replacing them with e.g. _?

We could also leave the space in, no warning, no replacement. It's a valid character on all platforms. We can still recommend more Java-like names, but we don't need to force it.

Alright, I just realized that I misread your statement in #6288:

Also reconsider the deprecation of space, as this caused several complaints.

Let's keep supporting spaces then. I'll change this PR.


Use libcrypto name from OPENSSL_CRYPTO_LIBRARY if set

Hello,
When building the SDK in shared libraries mode on linux, we provide the OPENSSL_SSL_LIBRARY and OPENSSL_CRYPTO_LIBRARY definition.
With aws sdk v1.9.157, that were no issue, the correct package was used and linked with.
With aws sdk v1.11, the linked is done with both the OPENSSL_CRYPTO_LIBRARY and the system file libcrypto.so.1
This is mainly because we changed the name of the built libcrypto by adding a suffix.
Note that we also rebuild the openssl code to use opensll 3.
So this PR may be of low interest for you.
Description of changes:
From log history, the file cmake/Findcrypto.cmake has been added for 1.9.235
I changed it to use the name from OPENSSL_CRYPTO_LIBRARY when setting crypto_SHARED_LIBRARY and crypto_SHARED_LIBRARY.
Check all that applies:

[X] Did a review by yourself.
[ ] Added proper tests to cover this PR. (If tests are not applicable, explain.)
[ ] Checked if this PR is a breaking (APIs have been changed) change.
[ ] Checked if this PR will not introduce cross-platform inconsistent behavior.
[ ] Checked if this PR would require a ReadMe/Wiki update.

Check which platforms you have built SDK on to verify the correctness of this PR.

[X] Linux
[ ] Windows
[ ] Android
[ ] MacOS
[ ] IOS
[ ] Other Platforms

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Another solution would be to directly use the variable if set:
diff --git a/cmake/Findcrypto.cmake b/cmake/Findcrypto.cmake
index 5425280d74..1ecc45dc21 100644
--- a/cmake/Findcrypto.cmake
+++ b/cmake/Findcrypto.cmake
@@ -39,21 +39,25 @@ else()
         PATH_SUFFIXES include
     )

-    find_library(crypto_SHARED_LIBRARY
-        NAMES libcrypto.so libcrypto.dylib
-        HINTS
-        "${CMAKE_PREFIX_PATH}"
-        "${CMAKE_INSTALL_PREFIX}"
-        PATH_SUFFIXES build/crypto build lib64 lib
-    )
+    if (OPENSSL_CRYPTO_LIBRARY)
+        set(crypto_SHARED_LIBRARY ${OPENSSL_CRYPTO_LIBRARY})
+        set(crypto_STATIC_LIBRARY ${OPENSSL_CRYPTO_LIBRARY})
+    else()
+        find_library(crypto_SHARED_LIBRARY
+            NAMES libcrypto.so libcrypto.dylib
+            HINTS
+            "${CMAKE_PREFIX_PATH}"
+            "${CMAKE_INSTALL_PREFIX}"
+            PATH_SUFFIXES build/crypto build lib64 lib
+        )

-    find_library(crypto_STATIC_LIBRARY
-        NAMES libcrypto.a
-        HINTS
-        "${CMAKE_PREFIX_PATH}"
-        "${CMAKE_INSTALL_PREFIX}"
-        PATH_SUFFIXES build/crypto build lib64 lib
-    )
+        find_library(crypto_STATIC_LIBRARY
+            NAMES libcrypto.a
+            HINTS
+            "${CMAKE_PREFIX_PATH}"
+            "${CMAKE_INSTALL_PREFIX}"
+            PATH_SUFFIXES build/crypto build lib64 lib
+        )
+    endif()

     if (NOT crypto_LIBRARY)
         if (BUILD_SHARED_LIBS)


Use FsSrGen package

ref #962

[X] replace FsSrGen.exe
[x] replace FsSrGen msbuild task
[x] remove temp fssrgen ci feed, use standard nuget feed

the FsSrGen use .NET 4.6 instead of .NET 3.5

@dotnet-bot test this please

@dotnet-bot test this please

ok, ready to review/merge.
the FsSrGen now use .NET 4.6 instead of .NET 3.5
I removed the FindUnusedResources.exe because was unused

in 0152d76 i removed $(FSharpSourcesRoot) because a build error on Proto.Compiler project, i think it doesnt have a FSharpSourcesRoot property

ok parametrized FsSrGen packages versions

Thanks @enricosada

@enricosada  I notice from GitHub that you have a dotnetcli built version of fsSrgen is there a nugget package containing that ?

@KevinRansom yes. I'll post tomorrow a coreclr branch with my wip dotnet cli build of fsharp.compiler.
The dotnet-fssrgen is in the myget feed
Fssrgen and fslexyac as dotnet cli tool are integrated in the dotnet build
I'll post fslexyac to master before that because @dsyme want to review it

Btw these packages are .net 4.6 but are built with dotnet cli, no fsproj


Report with Parameters

Upgraded to 18.1 from 16.2 and when I run a report in the web that has parameters, I get a null reference error in.
at Xpand.Persistent.Base.ModelAdapter.ModelSynchronizer2.GetModelValueInfos(IModelNode modelNode) in D:\Github\Fuzion\Xpand-Source\Xpand\Xpand.Persistent\Xpand.Persistent.Base\ModelAdapter\Synchronizers.cs:line 153    at Xpand.Persistent.Base.ModelAdapter.ModelSynchronizer2.ApplyValues(ModelNode node, Object component, PropertyDescriptorCollection properties) in D:\Github\Fuzion\Xpand-Source\Xpand\Xpand.Persistent\Xpand.Persistent.Base\ModelAdapter\Synchronizers.cs:line 95
at Xpand.Persistent.Base.ModelAdapter.ModelSynchronizer2.ApplyModel(IModelNode node, Object component, Action3 action) in D:\Github\Fuzion\Xpand-Source\Xpand\Xpand.Persistent\Xpand.Persistent.Base\ModelAdapter\Synchronizers.cs:line 79
at Xpand.Persistent.Base.ModelAdapter.ObjectModelSynchronizer.ApplyModel(Action`3 action) in D:\Github\Fuzion\Xpand-Source\Xpand\Xpand.Persistent\Xpand.Persistent.Base\ModelAdapter\Synchronizers.cs:line 32
at DevExpress.ExpressApp.Model.ModelSynchronizer.ApplyModel()

I cannot suggest anything without more info, since you only know your configuration do you think you can sample us?

Probably you relate to the #183

I created a sample project. It will create some dummy DomainObjects1s. There is an repx file in there. If you load that report, it has a parameter for Date >= ??
Run the report in the web and it will fire the error for you.
ReportParametersException.zip

By the way, I use Reports V1, not V2.

I changed my sample project and this error does not occur with ReportsV2.

to make it work in ReportsV1 you can simply disable the BAD controller. So In your can I would use this.
    public class Global : System.Web.HttpApplication {
        public Global() {
            InitializeComponent();
        }
        protected void Application_Start(Object sender, EventArgs e) {
            XpandModuleBase.DisableControllers(typeof(ASPxDateEditModelAdapter));

That works for now and gets me past my issue, thanks!


RAM leak problem when training in command batch_queue.get(). Where do you release resources once a batch is trained.

I have noticed that your training loop leaks small amounts of RAM memory.Any idea on what may have caused this?
time taken= 9.865329265594482  | steps= 1 | cpu= 51.8 | ram= 34.50078675328186 | gpu= [3101]
[5613]
time taken= 0.934636116027832  | steps= 2 | cpu= 27.0 | ram= 29.34866251942084 | gpu= [5613]
[3045]
time taken= 0.8695635795593262  | steps= 3 | cpu= 29.4 | ram= 29.217970957706278 | gpu= [3045]
[3021]
time taken= 0.8483304977416992  | steps= 4 | cpu= 29.8 | ram= 29.033316428574086 | gpu= [3021]
[2997]
time taken= 0.8630681037902832  | steps= 5 | cpu= 30.2 | ram= 28.87988403913803 | gpu= [2997]
[2997]
time taken= 0.8645083904266357  | steps= 6 | cpu= 29.4 | ram= 28.714746447210654 | gpu= [2997]
[2997]
time taken= 0.864253044128418  | steps= 7 | cpu= 29.3 | ram= 28.573093657739385 | gpu= [2997]
[2997]
time taken= 0.8693573474884033  | steps= 8 | cpu= 29.3 | ram= 28.389703885656044 | gpu= [2997]
[2997]
time taken= 0.8704898357391357  | steps= 9 | cpu= 29.4 | ram= 28.298690976454438 | gpu= [2997]
[2997]
time taken= 0.8670341968536377  | steps= 10 | cpu= 29.5 | ram= 28.13385097442091 | gpu= [2997]
[2997]
time taken= 0.8750414848327637  | steps= 11 | cpu= 29.5 | ram= 27.959884882309396 | gpu= [2997]
[2997]
time taken= 0.8624210357666016  | steps= 12 | cpu= 29.9 | ram= 27.784356443255188 | gpu= [2997]
[2997]
time taken= 0.8561670780181885  | steps= 13 | cpu= 29.8 | ram= 27.644241201568796 | gpu= [2997]
[2997]
time taken= 0.8609695434570312  | steps= 14 | cpu= 29.7 | ram= 27.51883186047002 | gpu= [2997]
[2997]
time taken= 0.8462607860565186  | steps= 15 | cpu= 29.7 | ram= 27.36641623650461 | gpu= [2997]
[2997]
time taken= 0.8624782562255859  | steps= 16 | cpu= 29.2 | ram= 27.23760941078441 | gpu= [2997]
[2997]
time taken= 0.8649694919586182  | steps= 17 | cpu= 29.4 | ram= 27.113514425050127 | gpu= [2997]
[2997]
time taken= 0.8661544322967529  | steps= 18 | cpu= 29.3 | ram= 27.004993310427178 | gpu= [2997]
[2997]
time taken= 0.8687705993652344  | steps= 19 | cpu= 29.8 | ram= 26.82090916192486 | gpu= [2997]
[2997]
time taken= 0.8823645114898682  | steps= 20 | cpu= 29.6 | ram= 26.688630454109777 | gpu= [2997]
[2997]
time taken= 0.8795809745788574  | steps= 21 | cpu= 29.4 | ram= 26.517987449146226 | gpu= [2997]
[2997]
time taken= 0.8857841491699219  | steps= 22 | cpu= 29.1 | ram= 26.40289455770082 | gpu= [2997]
[2997]
time taken= 0.8605339527130127  | steps= 23 | cpu= 29.5 | ram= 26.274509317663572 | gpu= [2997]
[2997]
time taken= 0.8524265289306641  | steps= 24 | cpu= 29.8 | ram= 26.16445065525575 | gpu= [2997]

my edit in main.py:
# with or without the given invalid mask
        if len(batch) == 4:
                img1, img2, flow, mask = [batch[i] for i in range(4)]
                train_log = pipe.train_batch(img1, img2, flow, geo_aug, color_aug, mask = mask)
        else:
                now = time.time()
                img1, img2, flow = [batch[i] for i in range(3)]
                train_log = pipe.train_batch(img1, img2, flow, geo_aug, color_aug)
                #print(batch_queue,data_queues,remove_queue)
                print("time taken=",time.time()-now," | steps=",steps,"| cpu=",psutil.cpu_percent(),"| ram=",psutil.virtual_memory().available * 100 / psutil.virtual_memory().total,"| gpu=",get_gpu_memory())

also, is there any way in which I don't have to load the full dataset on the memory for training in mxnet?

I have found that after every run of

batch = batch_queue.get()

an extra 0.10-0.15% ram is consumed using pdb.
(Pdb) print("| ram=",psutil.virtual_memory().available * 100 / psutil.virtual_memory().total)
 **ram= 28.66921519345203**
(Pdb) n
> /home/mask/maskflownet/MaskFlownet/main.py(572)<module>()
-> loading_time.update(default_timer() - t0)
(Pdb) print("| ram=",psutil.virtual_memory().available * 100 / psutil.virtual_memory().total)
**ram= 28.542640291935687**

I cannot find why this is happening but i am sure of it. Can you help me fix this please?

Hi vivasvan1, thanks for pointing out this problem.
We import this Queue method from the python queue package directly without any modification. I search on Google and find that other people encounter the same problem. So maybe this is not a problem with our code but the python queue package.


Include prereleases from downstream projects in pip release links page.

This fixes https://github.com/iree-org/iree/issues/19193.
Nightly releases of packages from https://github.com/iree-org/iree-turbine and https://github.com/nod-ai/shark-ai will now be available from https://iree.dev/pip-release-links.html with (for example)
pip install --pre --find-links https://iree.dev/pip-release-links.html iree-turbine

Previously, these projects relied on their own index pages like

https://github.com/nod-ai/shark-ai/releases/expanded_assets/dev-wheels
https://github.com/iree-org/iree-turbine/releases/expanded_assets/dev-wheels

While these links will continue to work, the consolidated page is more convenient for users.
Projects included
I'm starting with iree-turbine and shark-ai, since these are what I'm currently trying to simplify workflows for (example here). I left some notes in the workflow file too, but I could see us extending this to also include torch-mlir and stablehlo packages. The extra release scraping doesn't cost us much CI time, git storage on the gh-pages branch, or network bandwidth on the website. I also don't foresee any issues with making it easier to install nightly versions of those packages. I would pause a bit if the list grows too much beyond that though.
We could still switch to a PEP 503-compliant package index (usable with --extra-index-url, not -f, --find-links), like what PyTorch provides at https://download.pytorch.org/whl/cpu. Note the 50+ packages listed there, included mirrors of a few versions of all of PyTorch's dependencies.
Testing

Publish action: https://github.com/ScottTodd/iree/actions/runs/12188757072/job/34002585332

Pushed to GitHub pages: https://scotttodd.github.io/iree/pip-release-links.html

Installed using the release links:
python3.11 -m venv 3.11.venv
source 3.11.venv/bin/activate

pip install --pre --find-links https://scotttodd.github.io/iree/pip-release-links.html iree-turbine --no-deps
# Looking in links: https://scotttodd.github.io/iree/pip-release-links.html
# Collecting iree-turbine
# Downloading https://github.com/iree-org/iree-turbine/releases/download/dev-wheels/iree_turbine-3.1.0rc20241205-py3-none-any.whl (292 kB)
#   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 292.3/292.3 kB 8.1 MB/s eta 0:00:00
# Installing collected packages: iree-turbine
# Successfully installed iree-turbine-3.1.0rc20241205

Note on the shark-ai package
The shark-ai meta package (https://pypi.org/project/shark-ai/, https://github.com/nod-ai/shark-ai/tree/main/shark-ai) pins dependencies in ways that isn't currently compatible with this package installation flow. Specifically, the latest nightly releases pin shortfin==3.1.0 which can't be resolved by a release candidate like 3.1.0rc20241205, so this doesn't work as one might expect:
pip install --pre --find-links https://iree.dev/pip-release-links.html shark-ai
# Successfully installed shark-ai-3.0.0

# Test with an explicit version to demonstrate the issue:
pip install --pre --find-links https://scotttodd.github.io/iree/pip-release-links.html \
  shark-ai==3.1.0rc20241205
# ERROR: Could not find a version that satisfies the requirement shortfin==3.1.0 (from shark-ai) (from versions: 2.9.0rc20241108, 2.9.0rc20241109, 2.9.0rc20241110, 2.9.0rc20241111, 2.9.0, 2.9.1rc20241112, 2.9.1rc20241113, 2.9.1rc20241114, 2.9.1, 2.9.2rc20241115, 2.9.2rc20241116, 2.9.2rc20241117, 2.9.2rc20241118, 2.9.2, 3.0.0rc20241118, 3.0.0, 3.1.0rc20241119, 3.1.0rc20241120, 3.1.0rc20241121, 3.1.0rc20241122, 3.1.0rc20241123, 3.1.0rc20241124, 3.1.0rc20241125, 3.1.0rc20241126, 3.1.0rc20241127, 3.1.0rc20241128, 3.1.0rc20241129, 3.1.0rc20241130, 3.1.0rc20241201, 3.1.0rc20241202, 3.1.0rc20241203, 3.1.0rc20241204, 3.1.0rc20241205)
# ERROR: No matching distribution found for shortfin==3.1.0

I am not convinced that any non iree-org package should be included in the index. It might be reasonable to publish links for dependencies that are required but I don't know what those would be at the moment. The dependencies are available via PyPI and wheels like the ones provided via https://github.com/llvm/torch-mlir-release are not direct dependencies. Furthermore, it doesn't seem to be the right place to incclude downstream packages that make use of IREE. I would rather expect that those packages might include IREE in their index instead but here the other way around is proposed. I might miss the rational for this, so I am happy to discuss!

I'm fine with just including iree-org/iree-turbine in addition to this iree-org/iree repository in this PR and seeing how far that gets us.

I am not convinced that any non iree-org package should be included in the index. It might be reasonable to publish links for dependencies that are required but I don't know what those would be at the moment. The dependencies are available via PyPI and wheels like the ones provided via https://github.com/llvm/torch-mlir-release are not direct dependencies.

Here's my thinking for including other packages:

I consider the nightly releases in general to be a convenience we provide for developers and automated workflows, not a fully supported path that we want users to depend on. Most users should be using stable releases, and we need to keep the release train going on that. Our current SLO is a stable release every 6 weeks, but we could increase that frequency.
Including more packages increases the convenience factor, at little technical cost. The larger costs are social - does including a package in the index imply some association or support status?

Furthermore, it doesn't seem to be the right place to incclude downstream packages that make use of IREE. I would rather expect that those packages might include IREE in their index instead but here the other way around is proposed.

Sure, we could have each downstream project provide its own package index too, but I'd rather avoid the scenario where the nightly package install instructions are different depending on which downstream project(s) you are using.
We've had https://github.com/iree-org/iree/issues/9532 open highlighting the issue with having multiple package indices:
pip install \
  nodai-shark \
  -f https://github.com/nod-ai/SHARK/releases \
  -f https://github.com/llvm/torch-mlir/releases \
  -f https://github.com/nod-ai/shark-runtime/releases \
  --extra-index-url https://download.pytorch.org/whl/nightly/cpu`

That was two years ago, and today it looks something like this:
pip install \
  shortfin \
  -f https://github.com/nod-ai/shark-ai/releases/expanded_assets/dev-wheels \
  -f https://iree.dev/pip-release-links.html

We could publish a downstream https://nod-ai.github.io/shark-ai/pip-release-links that includes IREE nightly packages, but then other downstream projects may want to do the same, and it could end up fragmenting documentation, scripts, reproducer instructions, etc.
A more "final" fix would be to push pre-releases to the official package index on https://pypi.org/, but then we run into storage and bandwidth limits more quickly and the project release history looks messier. For example, see how stable releases are rc0/rc1/etc. candidates are clearly visible on https://pypi.org/project/tensorflow/#history compared to the stream of nightly releases on https://pypi.org/project/tf-nightly/#history.
Some related discussions in other parts of the Python packaging ecosystem, mostly around having a warehouse (pypi) API to set retention policies on pre-releases pushed there, avoiding the need to maintain project-specific package indices:

Geoffrey's analysis here: https://github.com/iree-org/iree/issues/9532#issuecomment-1262687021
https://discuss.python.org/t/publishing-nightly-builds-on-test-pypi-org-with-a-time-based-retention-policy/3152/9
https://github.com/pypi/warehouse/issues/8792

All that being said, my gut feeling is "eh, why not create this upstream index for any projects that use IREE to use?". There's a similar precedent in https://anaconda.org/scientific-python-nightly-wheels hosting 29 packages, many from different organizations like:

https://github.com/scipy/scipy
https://github.com/statsmodels/statsmodels
https://github.com/pandas-dev/pandas
https://github.com/contourpy/contourpy
etc.

If we were to do that, anchoring on IREE as the host makes sense to me, but I could see an argument for MLIR, LLVM, the Linux Foundation, or some other super-project / organization hosting.

Switched to just --repos="iree-org/iree,iree-org/iree-turbine" and updated the PR description.


Active Directory Authentication?

I love this dashboard - so does my leadership.
However, we have a requirement to secure these things behind Active Directory and restrict who can view them.  Preferably we'd authorized based on an AD group that all our DBA's and other relevant folks are members of, and anyone else would just get a restricted page or re-directed to our service portal.
However, since this runs using Flask, is there actually a way to implement AD authentication?

Hi Neil, you should be able to implement AD authentication by running flask using IIS-FastCGI:
https://medium.com/@bilalbayasut/deploying-python-web-app-flask-in-windows-server-iis-using-fastcgi-6c1873ae0ad8


Rename unsealer.image to bankVaults.image

Currently both the configurer and unsealer use the unsealer image config.
Rename it to bankVaults so that naming makes more sense.

Yes, it should.

I can help with this. :+1:

Appreciate it, thank you!

@sagikazarmark Should this be implemented only for https://github.com/bank-vaults/vault-helm-chart/blob/main/vault/values.yaml#L250-L254 or for the entire unsealer block i.e. https://github.com/bank-vaults/vault-helm-chart/blob/main/vault/values.yaml#L250-L276? The former is pretty straightforward.

Good question...
I'd say it would be ideal if we could have a common block for everything that's shared between the unsealer and the configurer, but I can see why that could prove challenging.
Maybe we could use templating to filter the relevant keys from the unsealer section and recursively merge it with a new bankVaults section. I think that should be possible using template functions.

Having an issue with creating a template for unsealer.args, will work on it for some time.

Hi @tanmay-pnaik, thanks for the interest, do you still need help with this? I can assist you and help out with setting everything up.

Hi @tanmay-pnaik, thanks for the interest, do you still need help with this? I can assist you and help out with setting everything up.

@ramizpolic That would be helpful. I'll raise a PR with my current changes and maybe we could then take it forward?


Windows: installs but isn't found?

Solution to issue cannot be found in the documentation.

[X] I checked the documentation.

Issue
On Windows (just a normal cmd.exe shell, not minGW or cygwin or WSL).
I can run conda install -c conda-forge fpm and it seems to install FPM.
> conda list | grep fpm
> fpm                       0.6.0                h57928b3_0    conda-forge

But, then I type fpm and get:
> 'fpm' is not recognized as an internal or external command,
operable program or batch file.

So the entry point isn't being installed or put in the right location or something? It looks like fpm.exe is being put in <miniconda installation>/bin. I notice there isn't anything else in there, and it doesn't seem to be in my path. I think most everything else goes into <miniconda installation>/Scripts. That's where I see other packages i have installed that have .exe files.
Installed packages
doesn't matter

Environment info
active environment : None
       user config file : C:\Users\blah\.condarc
 populated config files :
          conda version : 4.13.0
    conda-build version : not installed
         python version : 3.8.10.final.0
       virtual packages : __cuda=11.0=0
                          __win=0=0
                          __archspec=1=x86_64
       base environment : C:\Users\blah\Miniconda3  (writable)
      conda av data dir : C:\Users\blah\Miniconda3\etc\conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\blah\Miniconda3\pkgs
                          C:\Users\blah\.conda\pkgs
                          C:\Users\blah\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\blah\Miniconda3\envs
                          C:\Users\blah\.conda\envs
                          C:\Users\blah\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.13.0 requests/2.28.1 CPython/3.8.10 Windows/10 Windows/10.0.19042
          administrator : False
             netrc file : None
           offline mode : False

We do test for fpm --version running in the recipe, %PREFIX%\bin is supposedly in the path there. I'll try to rebuild and installing fpm to %LIBRARY_BIN%.


fix: enable custom model.

we only support the selectable items right now. but we need to support the user inputting the model like this:

Just tried this. It "worked" at first, by showing the Model as the one I entered into config.toml: "ollama/deepseek-coder:instruct"
However, I cleared the Model in the UI by accident, and now there's no way to get it working again. The config.toml still says "ollama/deepseek-coder:instruct" but I can't enter this manually into the UI (and it doesn't get recognized from the toml)

@lowlyocean does it work again if you close both backend and frontend, and restart?

Unfortunately not. I also cleared Local Storage, which makes it at least display the expected model at least - but the backend still thinks gpt3 model is selected

make build failed:
(opendevin) agent@DESKTOP-OJHF2BM:~/OpenDevin$ make build
Building project...
Checking Python installation...
Python 3 is already installed. Version: 3.11.8
Checking npm installation...
npm is already installed. Version: 10.5.0
Checking Docker installation...
Docker is already installed. Version: 20.10.12,
Installing Python dependencies...
Poetry is already installed.
Pulling Docker image...
Using default tag: latest
latest: Pulling from opendevin/sandbox
Digest: sha256:486e3da1d7c8ec54a2e3269071eed9b91bc75b82ca5d1ac1760d33883c39e575
Status: Image is up to date for ghcr.io/opendevin/sandbox:latest
ghcr.io/opendevin/sandbox:latest
Installing dependencies from lock file
No dependencies to install or update
Installing the current project: opendevin (0.1.0)
Activating Poetry shell...
Setting up frontend environment...
Detect Node.js version...
Current Node.js version is 21.7.1, corepack is supported.
Installing frontend dependencies with npm...
npm notice
npm notice New patch version of npm available! 10.5.0 -> 10.5.1
npm notice Changelog: https://github.com/npm/cli/releases/tag/v10.5.1
npm notice Run npm install -g<EMAIL_ADDRESS>to update!
npm notice
npm ERR! Cannot read properties of null (reading 'edgesOut')
npm ERR! A complete log of this run can be found in: /home/agent/.npm/_logs/2024-04-07T00_44_28_358Z-debug-0.log
make: *** [Makefile:38: build] Error 1
log-------------------------------------------------------------------------------------
138 verbose stack TypeError: Cannot read properties of null (reading 'edgesOut')
138 verbose stack     at #loadPeerSet (/home/agent/.nvm/versions/node/v21.7.1/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:1313:38)
138 verbose stack     at async #buildDepStep (/home/agent/.nvm/versions/node/v21.7.1/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:924:11)
138 verbose stack     at async Arborist.buildIdealTree (/home/agent/.nvm/versions/node/v21.7.1/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:203:7)
138 verbose stack     at async Promise.all (index 1)
138 verbose stack     at async Arborist.reify (/home/agent/.nvm/versions/node/v21.7.1/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/reify.js:154:5)
138 verbose stack     at async Install.exec (/home/agent/.nvm/versions/node/v21.7.1/lib/node_modules/npm/lib/commands/install.js:153:5)
138 verbose stack     at async module.exports (/home/agent/.nvm/versions/node/v21.7.1/lib/node_modules/npm/lib/cli-entry.js:61:5)
139 verbose cwd /home/agent/OpenDevin/frontend
140 verbose Linux <IP_ADDRESS>-microsoft-standard-WSL2
141 verbose node v21.7.1
142 verbose npm  v10.5.0
143 error Cannot read properties of null (reading 'edgesOut')
144 verbose exit 1
145 timing npm Completed in 17512ms
146 verbose unfinished npm timer reify<PHONE_NUMBER>585
147 verbose unfinished npm timer reify:loadTrees<PHONE_NUMBER>625
148 verbose unfinished npm timer idealTree:buildDeps<PHONE_NUMBER>414
149 verbose unfinished npm timer idealTree:node_modules/.pnpm/@monaco-editor+react@4.6.0_monaco-editor@0.47.0_react-dom@18.2.0_react@18.2.0/node_modules/@monaco-editor/react<PHONE_NUMBER>767
150 verbose code 1
151 error A complete log of this run can be found in: /home/agent/.npm/_logs/2024-04-07T00_44_28_358Z-debug-0.log

@zhonggegege Inside the opendevin project directory there's frontend and in it, node_modules. Please delete node_modules, then try again. This is unrelated to this PR, it's more that the frontend needs to be rebuilt from scratch (ish).
If it still doesn't work, please make a new issue.


[Documentation Bug]: Fix misspelling in README.md

Describe the bug ✍
Describe your bug here
Provide step by step information reproduce the bug 📄
Description
Select program in which you are contributing
Other
Code of Conduct

[X] I follow CONTRIBUTING GUIDELINE of this project.

Hey @ahmed0saber !
Thank you for raising an issue 💗
You can self assign the issue by commenting /assign in comment 😀
Make sure you follow CODE OF CONDUCT and CONTRIBUTING GUIDELINES 🚀
Don’t Forget to ⭐ our GameZone🎮
Make sure you join our Discord🕹

Hey @ahmed0saber ! Thank you so much for your raising the issue💗
It’s all yours, you can come anytime again and make some contributions! 🚀
Alone, we can do little, but together we can do so much! 😇


pull request for e2e test at 12/05/2016 07:15:03

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

optest/docs/updated_at.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Create Settings Page

The settings page is visible to users when they are logged in. The settings page will allow users to:

Change password
Select which notifications to keep.

UI should be clean and minimalistic. also, create a server action for changing passwords and notification settings updates.
Different types of notifications can be:

Inbox
Order Message
Order Updates
Buyer Brief

Hey assign this to me


Adding onaddstream shim

Description
Adding onaddstream shim based on ontrack
Purpose
ontrack is defined in WebRTC spec, onaddstream is not

@youennf Thanks for the contribution. Could you please sign the CLA? Will take a look after then, thanks!

Ping

can you rebase and force-push please? That should make travis happy

Ow was a bit fast there, it would be great if you do a followup PR with a test covering this shim.


Examples of use in Vue / docsify

Unfortunately, the codapi-js is not designed for situations where deferred rendering occurs, and large delay values cannot always help. There is no possibility of delayed initialization with binding to events yet.
Perhaps I'm missing something, so good examples would help significantly. So it would be extremely helpful to see usage examples in Vue/docsify.
P.S. Currently, for my tasks, I abandoned the widget in favor of my own Vue component(built into docsify) which directly sends a request to the codapi backend.

Unfortunately, I am not familiar with Vue and have no intention of learning it at this time. Could you please provide a Vue-free example of the behavior you'd like the widget to support?

I understand you perfectly and in any case, thank you very much for such a cool thing.
The problem with using codapi-js is not so much on the Vue plane, but rather in the way the widget is initialized. The delay approach is not always a good practice; it would be much more convenient to initialize on some event.
I’ll give an agnostic example that is not tied to Vue to understand the idea.
    // Add an event listener for the AfterMarkdownRendered event, which fires when the initial HTML document has been completely loaded and parsed
    document.addEventListener('AfterMarkdownRendered', function() {
      // Call the init() method of the codapi-js library after the DOM has been fully loaded and rendered as MD
      codapi.init();
    });

Obviously, the AfterMarkdownRendered event may be different in different libraries, and most likely it may not be associated with DOMContentLoaded directly as for plain HTML.
IMPORTANT that now the init() event is taken as an example, and there may be different cases, but if the widget had a public API, there would probably be different initialization levels (initAll() / init())
The technique for Vue 3 could be something like this:
    // Create a new Vue app
    const app = Vue.createApp({
      data() {
        return {
          markdownContent: ''
        }
      },
      mounted() {
        // Call the init() method of the codapi-js library after the Vue app has been mounted
        codapi.init();
      }
    });

    // Mount the Vue app to the #app element
    app.mount('#app');

We create a new Vue app and define a data property called markdownContent to store the Markdown content.  Then iInside the mounted lifecycle hook, we call the init() method of the codapi-js library to initialize it. This ensures that the library is initialized after the Vue app has been mounted. Finally, we mount the Vue app to the #app element using app.mount('#app'). The Markdown content is rendered inside a
 element using v-html directive, which allows us to dynamically bind and render HTML content from the markdownContent data property.
For docsify, we can implement this via custom js.
window.$docsify = {
  // ...
  mounted() {
    codapi.init();
  },
  // ...
};

In the index.html file, include the custom.js file after including the Docsify script:

Now, when the page finishes rendering the Markdown content to HTML, the codapi.init() function will be called automatically.
Of course, these are crutches for now, but perhaps in the future a more graceful way will appear.
Requests/suggestions/corrections are welcome.

Thanks for the detailed explanation! So basically what is required from codapi-js in this case is to export a proper init function. And this init should initialize all widget instances present on the page. Right?

Thanks for the detailed explanation! So basically what is required from codapi-js in this case is to export a proper init function. And this init should initialize all widget instances present on the page. Right?

I think that's exactly it. Actually, it is important not to forget to provide an example in the documentation. As a general case, you can mention any markdown deferred rendering systems, you don’t have to refer to Vue / docsify.
BTW I'll try to help with testing and give feedback.

Thanks a lot! I'll get back to you when I have something to show. Hopefully next week.

Before refactoring the code, I tried to reproduce the problem with docsify, but everything seems to work fine, even without using init-delay. Here is what I did:

Installed docsify:

mkdir hello-docsify
cd hello-docsify
npm i docsify-cli
npm exec docsify init ./docs

Added Codapi to index.html:

<!DOCTYPE html>

    <link rel="stylesheet"<EMAIL_ADDRESS>    <link rel="stylesheet" href="https://unpkg.com/<EMAIL_ADDRESS>/>


[rlgl] Shader sampler2D uniforms assignation is confusing

If you need two textures in a shader, theoretically you should do it this way:
model.material.maps[0].texture = textureA; // 0 is index for MAP_DIFFUSE
model.material.maps[1].texture = textureB; // 1 is index for MAP_SPECULAR

But in the fragment shader, defining these uniforms will not work correctly:
uniform sampler2D texture0;
uniform sampler2D texture1;

That's because texture1 is used for the location of MAP_NORMAL (index 2), while texture2 is used for MAP_SPECULAR (index 1). Therefore, the correspondence between material map indexes and uniform variable names doesn't seem consistent in theses two cases:

material.maps[1] ---> texture2
material.maps[2] ---> texture1

Once you know it, it's nothing serious, but I had to take a look to the source to figure it.
Not sure if this is the expected behavior, but I found it somewhat confusing.

I think the issue you mean is corrected in commit https://github.com/raysan5/raylib/commit/25ceec9b8f8f414b995469a24ade69725ead4fdb
Actually, current shaders system is prepared to deal with any lcoation name find in the custom shader, I mean, when loading a custom shader, texture location points should be set by the user this way:
model.material.shader.locs[LOC_MAP_DIFFUSE] = GetShaderLocation(model.material.shader, "texture0");  // Choose any name you use on the shader for texture used as diffuse
model.material.shader.locs[LOC_MAP_SPECULAR] = GetShaderLocation(model.material.shader, "texture1"); // Choose any name you use on the shader for texture used as specular/metalness
model.material.shader.locs[LOC_MAP_NORMAL] = GetShaderLocation(model.material.shader, "texture2");   // Choose any name you use on the shader for texture used as normal

When loading any shader, raylib tries to set some default location points (SetShaderDefaultLocations()) looking for some default names on the shader:
// Get handles to GLSL uniform locations (vertex shader)
shader->locs[LOC_MATRIX_MVP]  = glGetUniformLocation(shader->id, "mvp");
shader->locs[LOC_MATRIX_PROJECTION]  = glGetUniformLocation(shader->id, "projection");
shader->locs[LOC_MATRIX_VIEW]  = glGetUniformLocation(shader->id, "view");

// Get handles to GLSL uniform locations (fragment shader)
shader->locs[LOC_COLOR_DIFFUSE] = glGetUniformLocation(shader->id, "colDiffuse");
shader->locs[LOC_MAP_DIFFUSE] = glGetUniformLocation(shader->id, "texture0");
shader->locs[LOC_MAP_SPECULAR] = glGetUniformLocation(shader->id, "texture1");
shader->locs[LOC_MAP_NORMAL] = glGetUniformLocation(shader->id, "texture2");

But those names could not be found, becoming the location -1, and letting the user set it manually like shown before.

I tried to do it like you say but I didn't get it work (probably my fault). Finally I ended doing something like this:
int MY_TEXTURE_INDEX = 0;
model.material.maps[MY_TEXTURE_INDEX].texture = myTexture;
SetShaderValuei(model.material.shader, GetShaderLocation(model.material.shader, "myTextureSampler"), MY_TEX_INDEX, 1);

Gracias por tu ayuda!

Mmmph... it should work this way:
Model model = LoadModel("my_model.obj");     // Load model mesh and init default material with default shader

// Load a custom shader for the model
model.material.shader = LoadShader("custom.vs", "custom.fs"); // Load custom shader, tries to initialize default location points with default names

// Setup shader location points for custom shader sampler2D names
model.material.shader.locs[LOC_MAP_DIFFUSE] = GetShaderLocation(model.material.shader, "texture_dif");  // Set shader location for diffuse map name: texture_dif
model.material.shader.locs[LOC_MAP_SPECULAR] = GetShaderLocation(model.material.shader, "texture_spec"); // Set shader location for specular map name: texture_spec
model.material.shader.locs[LOC_MAP_NORMAL] = GetShaderLocation(model.material.shader, "texture_norm");   // Set shader location for normal map name: texture_norm

// Load textures that will be sent to corresponding shader locations (MAP_DIFFUSE ---> LOC_MAP_DIFFUSE)
model.material.maps[MAP_DIFFUSE].texture = LoadTexture("diffuse.png");
model.material.maps[MAP_SPECULAR].texture = LoadTexture("specular.png");
model.material.maps[MAP_NORMAL].texture = LoadTexture("normal.png");

Thanks again Ray, I will try it again as soon as I have some time for it!

Closing this issue. Feel free to reopen it if proposed solution doesn't work properly.


chore: release 1.0.2

PR checklist

[ ] This comment contains a description of changes (with reason).
[ ] If you've fixed a bug or added code that should be tested, add tests!
[ ] If you've added a new tool - have you followed the pipeline conventions in the contribution docs
[ ] If necessary, also make a PR on the nf-core/fastquorum branch on the nf-core/test-datasets repository.
[ ] Make sure your code lints (nf-core lint).
[ ] Ensure the test suite passes (nf-test test main.nf.test -profile test,docker).
[ ] Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
[ ] Usage Documentation in docs/usage.md is updated.
[ ] Output Documentation in docs/output.md is updated.
[ ] CHANGELOG.md is updated.
[ ] README.md is updated (including new tool citations and authors/contributors).

Although I see @MatthiasZepper is already involved in looking at that.

Although I see @MatthiasZepper is already involved in looking at that.

Yes, the botched Seqera Containers introduction is keeping me up on my toes. But the error here was nicely troubleshooted, pinpointed and reported by @znorgaard, so I could fix it. The download will work, once this PR is merged to tools and already does, when the action from the dev branch is used instead the version from master.
I do really not know how to proceed, if one doesn't want to wait for the tools update. I think, in this case, it would be fine to merge with a failing test. Otherwise, it is a hen and an egg problem: To merge dev to master, you need to pass the test version on master, that will be corrected with the merge from dev.

Happy to merge with the failed test then, if it'll be fixed anyway. Although that'll require someone with more permissions than me.

@znorgaard: nf-core download fix was reviewed and merged earlier today, so the test is now passing. Currently, on dev (sufficient for making the test pass), but there is a 3.1 release of nf-core tools expected some when later this week, so ordinary people will also be able to successfully download the pipeline.
Happy fastquorum releasing!


add authorizerPayloadFormatVersion to the authorizer

Adds the authoriser payload format version to the authoriser. This changes should only affect apigateway v2

Codecov Report

Merging #5144 (bb81083) into master (dbcee3c) will decrease coverage by 0.00%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #5144      +/-   ##
==========================================
- Coverage   95.89%   95.88%   -0.01%
==========================================
  Files         680      680
  Lines       67416    67404      -12
==========================================
- Hits        64646    64630      -16
- Misses       2770     2774       +4

Flag
Coverage Δ

servertests
36.86% <0.00%> (-0.01%)
:arrow_down:

unittests
95.82% <100.00%> (-0.01%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

moto/apigatewayv2/responses.py
100.00% <100.00%> (ø)

moto/resourcegroupstaggingapi/models.py
82.30% <0.00%> (-1.48%)
:arrow_down:

moto/route53/responses.py
96.69% <0.00%> (-0.42%)
:arrow_down:

moto/sns/models.py
96.83% <0.00%> (-0.40%)
:arrow_down:

moto/ssm/models.py
94.53% <0.00%> (-0.11%)
:arrow_down:

moto/core/models.py
96.39% <0.00%> (-0.04%)
:arrow_down:

moto/sqs/models.py
96.92% <0.00%> (-0.01%)
:arrow_down:

moto/config/models.py
99.34% <0.00%> (-0.01%)
:arrow_down:

moto/ses/feedback.py
100.00% <0.00%> (ø)

moto/moto_api/__init__.py
100.00% <0.00%> (ø)

... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dbcee3c...bb81083. Read the comment docs.

Hi @calvernaz, do you remember the usecase for this PR? I'm not able to reproduce this, i.e. creating a authorizer against AWS without version does no automatically assign one.
Does AWS add the version only for a specific parameter combination, maybe?
The request that I used:
{
    "authorizerType": "JWT",
    "identitySource": [
        "$request.header.Authorization"
    ],
    "jwtConfiguration": {
        "audience": [
            "sth"
        ],
        "issuer": "..."
    },
    "name": "testauth"
}

And response from AWS:
{
    "authorizerId": "..",
    "authorizerType": "JWT",
    "identitySource": [
        "$request.header.Authorization"
    ],
    "jwtConfiguration": {
        "audience": [
            "sth"
        ],
        "issuer": "..."
    },
    "name": "testauth"
}

The Terraform tests for TestAccAPIGatewayV2Authorizer also fail because of this change - they pass when I revert it.

Hi @bblommers, so it should come from here, so it should be only available for v2 APIs. It's also available here for CF.
I can't speak for why the terraform tests fail, but the setting is definitely there.
The use case comes from the lambda authorizers input and output format. (https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-lambda-authorizer.html),

Hi @calvernaz, do you remember the usecase for this PR? I'm not able to reproduce this, i.e. creating a authorizer against AWS without version does no automatically assign one. Does AWS add the version only for a specific parameter combination, maybe?
The request that I used:
{
    "authorizerType": "JWT",
    "identitySource": [
        "$request.header.Authorization"
    ],
    "jwtConfiguration": {
        "audience": [
            "sth"
        ],
        "issuer": "..."
    },
    "name": "testauth"
}

And response from AWS:
{
    "authorizerId": "..",
    "authorizerType": "JWT",
    "identitySource": [
        "$request.header.Authorization"
    ],
    "jwtConfiguration": {
        "audience": [
            "sth"
        ],
        "issuer": "..."
    },
    "name": "testauth"
}

The Terraform tests for TestAccAPIGatewayV2Authorizer also fail because of this change - they pass when I revert it.

The setting is definitely there, but I'm just curious at under which conditions it defaults to 2.0. It doesn't always take that default value.

The setting is definitely there, but I'm just curious at under which conditions it defaults to 2.0. It doesn't always take that default value.

My understanding is that it defaults to 2.0, for HTTP APIs, when there is no value. The documentation doesn't state that, but it says "if you need REST compatibility use 1.0". I think that's what serverless defaults to.

Not specifying this value for type=REQUEST actually throws an error. It's not stated in the documentation, but running this command:
aws apigatewayv2 create-authorizer --api-id ... --authorizer-type "REQUEST" --identity-source "" --name testauth2 --authorizer-uri "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:0000:function:test/invocations"

returns this error from AWS:
An error occurred (BadRequestException) when calling the CreateAuthorizer operation: AuthorizerPayloadFormatVersion is a required parameter for REQUEST authorizer

So it will never default to anything. I'll try to create a PR shortly to mimic this error in Moto, and reverse the default value.

Not specifying this value for type=REQUEST actually throws an error. It's not stated in the documentation, but running this command:
aws apigatewayv2 create-authorizer --api-id ... --authorizer-type "REQUEST" --identity-source "" --name testauth2 --authorizer-uri "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:0000:function:test/invocations"

returns this error from AWS:
An error occurred (BadRequestException) when calling the CreateAuthorizer operation: AuthorizerPayloadFormatVersion is a required parameter for REQUEST authorizer

So it will never default to anything. I'll try to create a PR shortly to mimic this error in Moto, and reverse the default value.

Thanks for getting to the bottom of this, clearly an oversight from my side.


Rewritten evaluation, added benchmarking functionality to eval and infer

Changes in this pull request:

Fixed #106 (unfortunately only in the case when data size is specified. Seems to be impossible to do it in the other case with the current tf.data API).
Fixed #107 in a sense that we don't expect any specific data format anymore and inference passes sample ids directly to the infer/finalize_inference functions to enable multi GPU (only implemented in speech2text so far).
Added benchmarking objects to inference and evaluation (had to fix get_num_objects_per_step to ensure that it is only called inside compile function of the model, since no tensorflow graph modification should happen after that).
Added convenient collect_if_horovod MPI wrapper, that will collect data when on horovod and do nothing otherwise.

@okuchaiev, can you test this PR on your classes, to see that they work (especially interested in data layer that has dynamic batch size)?


refactor(cmd-api-server): add socket.io connection to each ledger-connector

Is your feature request related to a problem? Please describe.
On the current codes, cmd-api-server does not have socket.io connection to each ledger-connector.  This communication channel is needed for developing the APIs of sendSyncRequest, sendAsyncRequest, and startMonitor as these are developed in cmd-socket-server via socket.io connection to each ledger.
This feature was already developed in cmd-socket-server, but in case of merging those codes as #1502, we need to develop it.
Describe the solution you'd like
Add the socket.io connection to cmd-api-server
EPIC
#1605

@takeutak Could you please expand on this part? On the current codes, cmd-api-server does not have socket.io connection to each ledger-connector. Support for socket.io was added a long time ago.

@petermetz Thanks for your advice!
As I received your advice, I looked the codes again.
Let me clarify one thing, is the following figure right for calling your mentioned codes?

figure: https://github.com/hyperledger/cactus/blob/main/packages/cactus-plugin-ledger-connector-besu/docs/architecture/images/run-transaction-endpoint.png
API client: https://github.com/hyperledger/cactus/blob/d5e1708e84ab5192d07410d5120e144d477ef6ce/packages/cactus-plugin-ledger-connector-besu/src/main/typescript/api-client/besu-api-client.ts
APi server: Is it right? > https://github.com/hyperledger/cactus/blob/main/packages/cactus-cmd-api-server/src/main/typescript/api-client/api-server-api-client.ts
Ledger connector: https://github.com/hyperledger/cactus/blob/9022064e245a92f71d2d303d77bfdaf64d1b1678/packages/cactus-plugin-ledger-connector-besu/src/main/typescript/plugin-ledger-connector-besu.ts

According to the figure, BLP sends requests to API clients on each ledger connector via socket.io connection (*1), and API clients sends requests to ledger connector via socket.io connection (*2), is it right?
(*1) BLP <--(socket.io)--> API clients on each ledger connector
(*2) API clients on each ledger connector <--(socket.io)--> ledger connector
I think your implementation is good, but it makes more useful to prepare the following ways:

abstraction of calling API client on each ledgers when (*1) is executed.  I think Verifier structure resolve the feature.
extension of socket.io connection structure to other ledgers than besu (Currently, socket.io connections seem to be developed only on besu connector in your implementation)

To be clear, this issue is not intended to remove the features you already developed.  It is only intended to enhance the features.

@petermetz Please review this implementation idea and share your comments (especially on preferred location of new components - SocketIOApiClient and Verifier)
The general idea is to divide the current Verifier from cmd-socketio into two components that will be more compatible with api-server architecture.
SocketIOApiClient

Will abstract the communication with any socketio connector (validator).
Will use common interface with currently existing BesuApiClient (socketio part), and any future OpenAPI connectors with socketio support.
In opposite to BesuApiClient, SocketIOApiClient will work with any socketio validator (but not with openapi-socketio ones - like besu, this can be a little misleading but I don't see a way around it at the moment).
Most of the implementation will be taken from Verifier without much change in implementation (I'd prefer making future refactors in separate PR for better change tracking, even if they seem urgent).
The only major change in SocketIOApiClient will be watchBlocksV1 function that will return Observable (to match BesuApiClient).
Not sure where to place it yet. I didn't find a place where it would fit perfectly, and adding new package seems a little too much for so little logic in this component. I'm thinking about packages/cactus-api-client/src/main/typescript/socketio-api-client.ts (next to Verifier).

Verifier (ledger monitor proxy)

Will take compatible ledger ApiClient as an ctor argument (at the moment only SocketIOApiClient and BesuApiClient).
I want to put it in packages/cactus-api-client/src/main/typescript/Verifier.ts (similar intent as described below)
Just like current socketio Verifier it will:

Reuse conenctions to same ledger connector hosts.
Keep track of monitoring process.
Expose callback based startMonitor/stopMonitor

In the future it could be integrated into ApiClient (packages/cactus-api-client/src/main/typescript/api-client.ts), because they seems to have a similar goal - "providing additional functionality to the DefaultApi". The problem - ApiClient should work both on the front and backend (already used in Angular app in one example), so there would be some work needed to make socketio communication platform independent. Also - more API discussion is needed, so this is topic for another issue/PR.

While implementing this I'll try to be api-server native (keeping https://github.com/hyperledger/cactus/issues/1502 in mind). I'll try to use this common code implementation in cmd-socketio to prevent code duplication, but it will be lower priority for me (if that's OK).
Here is the UML draft for new Verifier:

@outSH Thank you for the details. How will the ISocketApi deliver the mentioned abstraction to be ledger-agnostic when it comes to the WatchBlocksV1Progress type? E.g., what will that type look like that somehow works with all ledgers that we added/will add in the future to the list of supported ones?

@petermetz I don't have a definite plan for that yet, I'll experiment and see what fits the best. I was thinking about replacing WatchBlocksV1Progress to unknown in the interface (since Verifeir doesn't really care about the block structure), but for more type safety I'm now thinking about making ISocketApi generic on block type. Verifier'd then infer (with conditional type) the block type and use it in it's declarations (so passing BesuAPI would require onEvent() callback accepting only besu block class) - but it's just a rough idea at the moment, will see how it works in practice when I implement it.
Do you have any other comments regarding this feature?

@outSH

I don't have a definite plan for that yet, I'll experiment and see what fits the best.

More details would be very welcome as they come along. The reason I'm highly interested in the details of this specifically because the "common abstraction" seems to be the foundational premise for everything else that's in the issue.

I want to put it in packages/cactus-api-client/src/main/typescript/Verifier.ts (similar intent as described below)

Any chance that the filename conventions in place could be kept? (lower-hypen-case)

In opposite to BesuApiClient, SocketIOApiClient will work with any socketio validator (but not with openapi-socketio ones - like besu, this can be a little misleading but I don't see a way around it at the moment).

Calling it SocketIOApiClient would imply that there hasn't been SocketIO support added already a long time ago to the other one. Is there something else that refers to a real distinction that we could put in the name of it?

Is the Verifier#eventListenerHash somehow replicated or does this store state locally that breaks implicit horizontal scalability/auto-scaling in a production environment?

@petermetz @takeutak @izuru0
I've finished an initial implementation, published as draft PR here -> https://github.com/hyperledger/cactus/pull/1660. Please review and share your comments. There are some more things to do, but the main logic is already there.
@petermetz

More details would be very welcome as they come along. The reason I'm highly interested in the details of this specifically because the "common abstraction" seems to be the foundational premise for everything else that's in the issue.

In the end ISocketApi (renamed to ISocketApiClient) uses generic type to work with any ledger block/event data format (see how besu-api-client implements ISocketApiClient). This can be extended to use generics for other specific parts of the interface like in IPluginLedgerConnector. I'm also thinking about splitting this interface into smaller interfaces (suitable for mixin).

Any chance that the filename conventions in place could be kept? (lower-hypen-case)

Yes, of course, it was a copy-paste typo.

Calling it SocketIOApiClient would imply that there hasn't been SocketIO support added already a long time ago to the other one. Is there something else that refers to a real distinction that we could put in the name of it?

Yes, I'm not satisfied with the name either, ( but unfortunately couldn't think of anything more suitable, I'm very open to suggestions :) It's meant to work with current, non-openapi based socketio interfaces, maybe we could add Generic, Common or Legacy prefix to it, or something else, but I'm afraid the name would be too mouthful. Maybe ValidatorApiClient?

Is the Verifier#eventListenerHash somehow replicated or does this store state locally that breaks implicit horizontal scalability/auto-scaling in a production environment?

I made a change and Verifier has a one-to-one relation to the ApiClient now. eventListenerHash has been replaced with runningMonitors, which are just named subscriptions to watchBlock subject.

@outSH Got it, thank you for the adjustments so far. I'll head over to the draft PR and we can have a much more targeted discussion over there.

Yes, I'm not satisfied with the name either, ( but unfortunately couldn't think of anything more suitable, I'm very open to suggestions :) It's meant to work with current, non-openapi based socketio interfaces, maybe we could add Generic, Common or Legacy prefix to it, or something else, but I'm afraid the name would be too mouthful. Maybe ValidatorApiClient?

Based on my hope that we'll end up consolidating I would go with Legacy, that way it might be less confusing to people that there are still somehow to implementations of something that are seemingly doing the same thing.

I made a change and Verifier has a one-to-one relation to the ApiClient now. eventListenerHash has been replaced with runningMonitors, which are just named subscriptions to watchBlock subject.

Awesome, thank you for updating it like that!


[openimageio] add libheif as an optional dependency, remove libsquish as a dependency

Workaound for #29156, but also is a good feature. Additionally, libsquish is not a dependency as of https://github.com/OpenImageIO/oiio/commit/5e67c09ecede66cc26c05d7134934c1b92385e9a#diff-0d57038c2e7b5bf8660e3859a78c7e96fbdb6b77604d4510e137de90b9e8f9e7

[x] Changes comply with the maintainer guide
[x] The "supports" clause reflects platforms that may be fixed by this new version
[x] Any fixed CI baseline entries are removed from that file.
[x] Any patches that are no longer applied are deleted from the port's directory.
[x] The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
[x] Only one version is added to each modified port's versions file.

Thanks!


rubyfmt -i command fails with Ruby Tree Deserialization Error

Ruby version: 2.7.3
Rubyfmt git sha: homebrew release, see version check below

% rubyfmt --version
rubyfmt-main 0.8.0-pre

% brew list rubyfmt
/opt/homebrew/Cellar/rubyfmt/0.8.0/.crates.toml
/opt/homebrew/Cellar/rubyfmt/0.8.0/.crates2.json
/opt/homebrew/Cellar/rubyfmt/0.8.0/bin/rubyfmt
/opt/homebrew/Cellar/rubyfmt/0.8.0/bin/rubyfmt-main

Input file
I can't share my source code at the moment as it doesn't belong to me, but I could change the names of everything to generic names and just share the structure of it if that would help.
~Here is a general skeleton of what it looks like (the real file is around 60 lines long):~
Edit 2022-12-03: (Updated code sample)
Here is the real structure of the original source code with a bunch of names changed to anonymize:
# frozen_string_literal: true

module Fruit
  module Peach
    class Distribution
      attr_reader :baskets

      def initialize(baskets:)
        @baskets = baskets
        @modifiers = {}
      end

      def prioritized_baskets
        modify_baskets
        prune_baskets
        set_primary_basket
        baskets
      end

      private

        attr_reader :modifiers

        def modify_baskets
          baskets.each do |basket|
            baskets.contents.each do |item|
              modifier = find_modifier(item) || build_modifier(item)
              modifier.modify(baskets, item)
            end
          end
        end

        def build_modifier(item)
          modifiers[hash_item item] = Modifier.new(
            required_quantity: item.unit.required_quantity
          )
        end

        def find_modifier(item)
          modifiers[hash_item item]
        end

        def prune_baskets
          baskets.reject!(&:empty?)
        end

        def set_primary_basket
          baskets.first.is_primary = true if baskets.present? && baskets.none?(&:is_primary)
        end

        def hash_item(item)
          shipment = item.unit.shipment
          product  = item.unit.product

          return product.hash unless shipment

          product.hash ^ shipment.hash
        end
    end
  end
end

I found https://github.com/fables-tales/rubyfmt/blob/trunk/RELEASE_README.md and tried running the debug binary like so
% rubyfmt-debug -i -- <path-to-my-file>.rb
zsh: command not found: rubyfmt-debug

I also tried installing the separate binary from Homebrew like so
% brew install rubyfmt-debug
...
Warning: No available formula with the name "rubyfmt-debug". Did you mean rubyfmt?

Rubyfmt's output
% rubyfmt -i -- ./<path-to-my-file>.rb
Error! source: ./<path-to-my-file>.rb

!!! Ruby Tree Deserialization Error !!!

Rubyfmt failed to correctly deserialize a tree from ripper. This is a bug that needs to be reported.
File a bug report at https://github.com/penelopezone/rubyfmt/issues/new.
Ideally you would include the full source code of the program you ran rubyfmt with.
If you can't do that for some reason, the best thing you can do is rerun rubyfmt on this program
with the debug binary with `2>log_file` on the end and then send us the log file that gets generated.

This breaks because
rubyfmt should be able to execute against *.rb files without throwing a Ruby Tree Deserialization Error.
Question
How does one run the rubyfmt-debug binary - is it available from binary that is built from source, and not the binary that is installed via Homebrew?

Yes, the debug binary is only available when building from source, brew doesn't ship debug binaries. Are you sure that (1) the example you've provided still fails and that (2) this isn't fixed on the current master branch? The example you've provided here passes both on 0.8.0 and master as far as I can tell.

Ok thanks good to know about the homebrew releases not shipping with debug binaries.
I did briefly try the Github release just now but got a macOS system protection error:

“rubyfmt-debug” can’t be opened because Apple cannot check it for malicious software.
This software needs to be updated. Contact the developer for more information.

I can deal with that at a later date, but for now I was able to go back into the cloned repository, pull the latest code down, and execute the debug binary this way:
% target/debug/rubyfmt-main -i -- <path-to-file> 2>log_file

(Had to change the file extension to .txt afterward so I could upload it)
I've attached that log file to this comment:
log_file.txt
The only thing I changed was the directory names and file names on the one line where it says Error! source: <filepath>
I will also update my sample to be more accurate as after thinking about it more today, the sample I provided is not that valuable as it's too far away from the real code. I will update the issue description with the real code structure and just anonymize it.

I've just updated the code sample in the issue description to be much closer to the real source code. I tested rubyfmt with the latest changes from Github and got the same result so it should be repeatable now. Sorry for providing the low-value code sample originally.

Thanks for updating, yeah the bit it fails on is modifiers[hash_item item] = .... A minimal repro is
foo[puts a] = ""

Will follow up on this later (or anyone else is welcome to take a look if you're interested!), but in the meantime this will work if you do modifiers[hash_item(item)] instead in all the places that happens.

Thank you for fixing that so quickly! I was able to pull down the latest code and it ran on that file successfully 👍


Non-existing snarkyjs commit referenced in xmas-sprint branch

See https://github.com/Trivo25/mina-zk-rollup/tree/ca01fbf1c8d547fd5a69f1befff67f2a6df82d99/src
https://github.com/o1-labs/snarkyjs/tree/129a35c387d750f661d8635b70e141a297496cd7 doesn't exist

Hey there, I use a slightly modified snarkyjs version that's not public at the moment, hence the commit leads into the void

@Trivo25 would you be able to push the commit, or at least explain the change? Thank you.

It mainly exposes a set of functionalities that are not natively exported by SnarkyJS, and since I am still actively working on those parts of the system (far from being completely done), I am actively changing things as I go, so that's why it isn't public yet

Thank you for the explanation.