Compare performance against synapse/dendrite

This media repo is intended to be the repo for multiple homeservers, therefore it is expected to see N times the load when compared to synapse/dendrite (where N is the number of homeservers the deployment is responsible for). This also needs to perform quickly so that clients don't get flack for having slow media (as most users aren't going to notice/care what is powering their media).
The results of this should be published somewhere and be updated regularly. Possible expansions would be to have a test on commits/PRs to verify the times aren't on an upward trend.
Off the top of my head, here's a few things to compare against synapse/dendrite:

Uploads (small/med/large) - random data to avoid dedupe
Uploads (small/med/large) - same data to encourage dedupe
Thumbnailing local media (small/med/large) - cold & warm cache
Downloading local media (small/med/large) - cold & warm cache
Downloading remote media (small/med/large) - cold & warm cache (ignore latency of remote server as there's not much we can do about that)
Thumbnailing remote media (small/med/large) - cold & warm cache (ignore latency of remote server)
Processing 10,000 concurrent uploads - cold start
Processing 10,000 concurrent local downloads - cold start
Processing 10,000 concurrent local thumbnails - cold start
Processing 10,000 concurrent remote downloads - cold start (ignore latency of remote server)
Processing 10,000 concurrent remote thumbnails - cold start (ignore latency of remote server)
Processing 10,000 concurrent random requests (uploads, download, thumbnail - local and remote - avoid and encourage dedupe) - attempt to cause caches to become useless and cold
... and other things to generally try and break the repo

Note: 10k concurrent requests may be ambitious. It should handle at least 1k before exhibiting symptoms of load

I haven't measured, but realistically this project isn't trying to be faster/better in every way. It's primary purpose is to deduplicate media and provide better administrative tooling for media, which will eventually make its way to the spec itself.
Performance is still important, though so far it appears to perform about the same. If it gets worse then we can take a look at it. Equally, if there's no bugs to fix then performance optimizations can be done to speed it up a bit (more in-memory caching, etc).


Template type not inferred correctly when using (ConcreteType|T)

This works: http://bit.ly/2ogmb6h (this shows the intention of using Error|T)
This shows an error: http://bit.ly/2ngvO3a (the difference is I no longer have a T in the param list but a Promise<T>)
found   : Promise<number>
required: Promise<(Error|number)>

On one hand, it looks like it fails at inferring the type of T. On the other hand, because of covariance, a promise returning a number IS-A special case of a promise that can return either number or Error.

Switching to CHECK_TYPES_NEW_INFERENCE gets rid of the error, however the AST is the same, it still shows the wrongly inferred T. Not sure if that is a bug in the debugger or a separate bug in NTI that it doesn't report the problem.

@brad4d I believe this has been fixed (certainly the error example no longer issues an error) so the issue should be able to be closed

Yes, this was probably fixed when we improved our type annotations for Promise a few months ago.
Thanks.


Do not understand what the output means

Forgive me if this question is too simple, but I cannot really understand what the output means from the command line (runs on linux and not have a GUI):
Flat Profile:
2.03 sun.reflect.DelegatingMethodAccessorImpl.invoke
Does this mean the "sun.reflect.DelegatingMethodAccessorImpl.invoke" method took 203% cpu time?
And
Tree Profile:
1.00 java.lang.Thread.run
0.00 java.util.concurrent.ThreadPoolExecutor$Worker.run
Tree profile makes me even more confused.

Hi,
Can you please send the HPL log for analysis?
From your output it would also seem you are not using latest version. Can you verify with latest?
Thanks,
Nitsan

Excellent Question @nitsanw - I was going to reply this today to ask the same question!

@nitsanw @RichardWarburton
I'm using the version downloaded from http://insightfullogic.com/honest-profiler.zip, and the unzipped files are marked from January 8, 2015.
The output file is too large to paste all of it here, so I keep the main structures and omit some of the contents and paste it as following:

/usr/lib/jvm/java-1.8.0-openjdk-<IP_ADDRESS>-2.b17.el7_1.x86_64/bin/
/usr/lib/jvm/java-1.8.0-openjdk-<IP_ADDRESS>-2.b17.el7_1.x86_64/bin//../lib/tools.jar
Printing Profile for: /tmp/logs/log.hpl
11:33:59.859 [main] DEBUG c.i.honest_profiler.core.Conductor - Reached end of log for FileLogSource{file=/tmp/logs/log.hpl} due to NOTHING
Number of stack traces: 154869

Flat Profile:
0.12 java.net.SocketInputStream.socketRead0
0.11 java.net.SocketOutputStream.socketWrite0
0.04 sun.misc.Unsafe.park
0.08 java.util.concurrent.locks.LockSupport.parkNanos
0.03 sun.reflect.Reflection.getCallerClass
0.03 sun.misc.Unsafe.unpark
0.02 java.lang.Object.wait
0.02 sun.nio.ch.EPollArrayWrapper.epollWait
0.02 sun.nio.ch.EPollArrayWrapper.interrupt
0.01 sun.nio.ch.FileDispatcherImpl.write0
0.01 java.net.PlainSocketImpl.socketAvailable
0.01 java.lang.String.intern

skipped ...

Tree Profile:
1.00 java.lang.Thread.run
0.00 java.util.concurrent.ThreadPoolExecutor$Worker.run
0.00 java.util.concurrent.ThreadPoolExecutor.runWorker

skipped ...

By the way - I've uploaded a new version of the binary at  http://insightfullogic.com/honest-profiler.zip which you might want to test.

hi @RichardWarburton
looks like the newer version requires "GLIBCXX_3.4.21" while I'm running on a CentOS 7.0 and not able to find it.
$ strings /usr/lib64/libstdc++.so.6 | grep GLIBCXX
GLIBCXX_3.4
GLIBCXX_3.4.1
GLIBCXX_3.4.2
GLIBCXX_3.4.3
GLIBCXX_3.4.4
GLIBCXX_3.4.5
GLIBCXX_3.4.6
GLIBCXX_3.4.7
GLIBCXX_3.4.8
GLIBCXX_3.4.9
GLIBCXX_3.4.10
GLIBCXX_3.4.11
GLIBCXX_3.4.12
GLIBCXX_3.4.13
GLIBCXX_3.4.14
GLIBCXX_3.4.15
GLIBCXX_3.4.16
GLIBCXX_3.4.17
GLIBCXX_3.4.18
GLIBCXX_3.4.19
GLIBCXX_DEBUG_MESSAGE_LENGTH

will try to compile GLIBCXX_3.4.21 on my CentOS 7. but it takes quite a lot effort. wish me luck.

Eurgh, someone else reported that problem as well. Let me investigate further.

@RichardWarburton thx, that would be awesome (and quite a burden release for me) to support lower version of GLIBCXX because upgrading it is a very tough quest after I did some google :)

So, after investigation I've concluded the simplest way for me to do this is to run a chroot or VM with an older GLIBCXX and compile the build on there. I'll try to get this setup soon.

@MichaelScofield I've uploaded a binary to http://insightfullogic.com/honest-profiler.zip which should work on as old as GLIBCXX_3.4.11.
I'll try and keep the build baselined to this version of GCC for as long as possible.
I'm also going to close this issue as I think its about an out of date UI being used. If the current UI is also confusing then please re-open.

OK, thanks @RichardWarburton :)


Request: Designate discord replies

In discord, you can right-click on a message and "reply" to it, linking context for the sake of knowing what someone is referring to.
I think the easiest implementation is;
[User on Discord] (In reply to earlier message) : I don't think that is quite the case.
The nicer one albeit would take a lot more work is;
[User on Discord] (Replying: first four words of message) : Well, maybe but I don't think so.

Implemented!


Run Jamaica

We are using JM as fixture test data (small-ish country with relative range of access and relatively good network data and population data).
I've started putting the data and now the point is to make it work, so it helps making it data agnostic, work with purely open data, and also get the results to build the tests.

[x] osm data for road network.
[x] check network connectivity.
[x] POIs to test (hospitals and banks).
[x] villages. ("village" tag).
[ ] population data. Partial Source here
[ ] get results.

WIP on Jamaica branch

This is pretty stale and doesn't use the upgraded code. I think we can close here, and reopen if relevant with the same checklist.


Docs: プライベートメンバーの先頭にはアンダースコアをつける

Tasks

[ ] Docs: プライベートメンバーの戦闘にはアンダースコアをつける

Writing the Issue

[ ] Write each task for this issue using checkboxes
[ ] Write all verbal instructions as tasks

Before Working on the Issue

[ ] Assign yourself
[ ] Share your screen on Discord
[ ] Write which issue you are starting on Slack
[ ] Git: Fetch the latest main
[ ] Git: Create a branch with the name of the issue

Pre-PR Checks

[ ] Insure that changes include only what is necessary for this PR
[ ] Implement necessary Unit tests
[ ] Implement necessary E2E tests
[ ] Perform functionality checks

https://github.com/sinProject-Inc/sinpro-dev/issues/60


added detection for browser/server into _app.tsx -- web3react couldnt…

This pull request addresses the Error: Invariant failed: A root already exists... error that is thrown when using yarn dev and references Issue #9

Did some digging and this looks to be caused from Next.js running the web3react code on the server. Adding a ternary to detect whether the code is running on browser or server-side resolves this. By detecting the browser, the code in _app.tsx that relies on web3react's client-side context only runs in the browser and not the server.
There is an open issue for this in the web3react repo and a similar fix was used, except with process.browser instead of typeof window !== 'undefined' - The Next.js team recommends against using process.browser
This change resolved the issue locally for me when using yarn dev and also when making live changes (seems to also resolve any hot reloading issues stemming from the SSR errors).
I'm not 100% confident about what should be returned if the browser isn't detected -- right now the ternary is returning null so if the server is detected instead of the browser nothing is returned. Returning <Component/> also works, but including <SiteLayout>... breaks it since that seems to rely on the context provided by web3react.

I'm not 100% confident about what should be returned if the browser isn't detected
I think an ideal state is returning a static site that Web3 hydrates client-side. Definitely requires some work to untangle some downstream dependencies.
@nateawelch thoughts on the achievableness of SSR that Web3 hydrates client-side?

I'm not 100% confident about what should be returned if the browser isn't detected

I think an ideal state is returning a static site that Web3 hydrates client-side. Definitely requires some work to untangle some downstream dependencies.
@nateawelch thoughts on the achievableness of SSR that Web3 hydrates client-side?

I think that would be ideal too. The static version would be returned instead of the null and then once the context is added from web3react it would fully hyrdate.

I'm not 100% confident about what should be returned if the browser isn't detected

I think an ideal state is returning a static site that Web3 hydrates client-side. Definitely requires some work to untangle some downstream dependencies.
@nateawelch thoughts on the achievableness of SSR that Web3 hydrates client-side?

Agreed that this would be the best path forward, and I don't see why it shouldn't be achievable - basically validating we're in a browser before fetching web3 context. I'm happy to work on this since I'm probably most familiar with the web3 portion, but @jonathanprozzi let me know if you want to take a crack at it

Agreed that this would be the best path forward, and I don't see why it shouldn't be achievable - basically validating we're in a browser before fetching web3 context. I'm happy to work on this since I'm probably most familiar with the web3 portion, but @jonathanprozzi let me know if you want to take a crack at it

I'm still building out my skills with web3 but I'm happy to provide any help with troubleshooting. I'm much more experienced with the Next.js side of things than web3.
The ternary checker for typeof window !== 'undefined' validates that we're executing in the browser. Before replacing with null I had passed in the original SiteLayout as a test, and it seems that there is something in there relying on the web3 context. I haven't had a chance to dig much deeper.
I've enjoyed working on this with you all and I'm happy to continue troubleshooting (especially from the Next.js side of things since that's where more of my experience is) but I don't want to delay due to needing to boost my web3 skills.

@jonathanprozzi I had a thought; what if we tie the typeof window !== 'undefined' check to environment. If it's local dev, disable SSR; if its production, let it ride. This way, prod keeps SSR and local can yarn dev.
https://docs.netlify.com/site-deploys/overview/
If it works, built a simple isProduction() method in utils, we'll need it. Add a doc block for the reasoning as well.

Sorry for the delay in response -- got completely swamped this week. This sounds like it could work too -- did you have a chance to test this out? Apologies for my lack of response. Just now getting caught up.

Sorry for the delay in response -- got completely swamped this week. This sounds like it could work too -- did you have a chance to test this out? Apologies for my lack of response. Just now getting caught up.

No worries; I put in PR-37 with a working concept. It does solve the local dev issue and in principal would work in production as well.

Awesome! Definitely looking forward to getting caught up on things this weekend.

Closing this in favor of the PR-48 dynamic import


Ignores parserOverride plugin option

Following, babeljs.io recommendation I forked @babel/parser and created my own to add some custom syntax. My source files are building and compiling fine after setting things up with a .babelrc.js file looking like:
const { parse } = require('../my-babel-fork/packages/babel-parser/lib');

module.exports = {
  plugins: [
    {
      parserOverride(code, opts) {
        return parse(code, opts);
      }
    }
  ]
};

However, babel-eslint seems to be completely ignoring the plugin option and keeps using the original @babel/parser instance. I know that for a fact because if a remove node_modules/@babel/parser and symlink that to my fork, it starts linting correctly, recognising new syntax and clearing all errors.
I also tried setting the parserOverride option directly inside parserOptions.babel to no avail.
Am I missing something? Is there any way to make this work? Thanks in advance!

Just noticed changes introduced in latest 11.0.0-beta.0 which seem to address my issue. Closing!


Identical alternates with different cigar strings

I recently got the following call line in an output vcf (truncated for simplicity):
chr5	112175675	.	AAGAGA	AA,AA	2.85084e-14	PASS	AB=0,0;ABP=0,0;AC=0,1;AF=0,0.5;AN=2;AO=0,0;CIGAR=1M2D1M,1M4D1M;DP=176

Both alternates are listed as AA, which causes problems with some external tools (vcftools in particular).  Shouldn't the cigar be consistent with the length of the reference (AAGAGA)?  Or should the alternates be different?

This is a bad bug. Could you share the input data that's causing this?
(Just a minimal bit of it that's enough to make these alleles show up.)
On Thu, Sep 3, 2015 at 7:06 PM, Erik Reckase<EMAIL_ADDRESS>wrote:

I recently got the following call line in an output vcf (truncated for
simplicity):
chr5    112175675   .   AAGAGA  AA,AA   2.85084e-14 PASS    AB=0,0;ABP=0,0;AC=0,1;AF=0,0.5;AN=2;AO=0,0;CIGAR=1M2D1M,1M4D1M;DP=176
Both alternates are listed as AA, which causes problems with some external
tools (vcftools in particular). Shouldn't the cigar be consistent with the
length of the reference (AAGAGA)? Or should the alternates be different?
—
Reply to this email directly or view it on GitHub
https://github.com/ekg/freebayes/issues/192.

Here's the sample data that reproduces the issue:
https://www.dropbox.com/s/c9ykhsjbig600k6/dupe_alternate_data.tar.gz?dl=0
There's a SAM in there and a targeted mutation vcf.  We do have two deletions listed in the targeted file at the same location, so that might be part of the problem.
freebayes -f <indexed hg19> --ploidy 2 --min-base-quality 20 --no-snps --variant-input adenomah.vcf.gz --only-use-input-alleles --min-mapping-quality 20 ecs.bam

Any update on this issue?


SVG export exports wrong part of the canvas

If canvas camera is moved (not at 0@0), SVG export does export wrong part of the canvas.

Code used in the example:
c := RSCanvas new.
boxes := (-200 to: 200 by: 100) collect: [ :pos | RSBox new color: Color random; translateBy: pos @ pos; yourself ].
c addAll: boxes.
c camera translateBy: 150@150.
c open.
c exportToSVG.

Resulting SVG file:

Looks like the exporter does need pixel/space transformations. It seems fixed by changing RSSVGExporter>>#visitCanvas:
Instead of
	pathTransform
		translateBy: (extent / 2) asFloatPoint;
		scaleBy: aRSCanvas camera scale asFloat;
		translateBy: aRSCanvas camera position negated asFloatPoint.

use
	pathTransform
		translateBy: (aRSCanvas camera fromPixelToSpace: extent / 2) asFloatPoint;
		scaleBy: aRSCanvas camera scale asFloat;
		translateBy: (aRSCanvas camera positionFromSpace: aRSCanvas camera position negated) asFloatPoint / aRSCanvas camera scale.

I am not sure it would work with other coordinate systems and possible transformations, but at least it fixes this specific problem.


chore: improve SQL parsing

SUMMARY

This is the first PR in a series to improve our SQL parsing, in effort to clean up the code base and make it more secure.
Currently, we have a class called superset.sql_parse.ParsedQuery that is used as an interface every time we need to parse or manipulate SQL. There are a few problems with the status quo:

Even though the class is called ParsedQuery, it's designed to accept a single SQL statement. For example, it has a set_or_update_query_limit method that works only on the first statement, if there are multiple.
Even though it's designed to accept a single statement, it doesn't raise an error when instantiated with a query that has multiple statements, so in several places the class is used with a full query.
There are multiple places that import sqlparse directly to manipulate SQL. For example, when inserting RLS into a query, or when modifying a query to insert a LIMIT or TOP.
In many places we manipulate SQL inefficiently, serializing and deserializing the SQL multiple times.

Ideally we'd have a single abstraction that handles SQL parsing. Superset code should never parse code directly by importing a 3rd party library like sqlparse, sqloxide, or sqlglot; instead, it should only call the abstraction. Similarly, the abstraction should wrap library specific exceptions (like sqlglot.errors.ParseError) with its own.
This PR introduces two new classes that aim to be those abstractions: SQLQuery and SQLStatement. The SQLStatement strictly represents a single SQL statement. These classes are instantiated with SQL and an optional engine (a BaseEngineSpec.engine attribute), and should provide all the methods needed for manipulating and introspecting SQL queries and statements. The initial implementation uses sqlglot, but the clean interface allows the parser to be changed in the future if we want.
My plan for improving the SQL parsing is first to get rid of code outside of sql_parse.py that using sqlparse directly:

[x] Introduce SQLQuery and SQLStatement, and replace any simple calls to sqlparse with calls to the new classes (this PR).
[ ] Implement a custom SQLStatement class for Kusto, since it doesn't use conventional SQL.
[ ] Implement insert_rls_in_predicate inside SQLStatement.
[ ] Implement get_cte_query inside SQLStatement.
[ ] Implement apply_top_to_sql in SQLStatement.

At this point we should have sqlparse being used only inside sql_parse.py by ParsedQuery. The last steps would be:

[ ] Replace all calls to ParsedQuery with calls to the new classes.
[ ] Get rid of the sqlparse dependency.

Note that despite the big changes, no public interfaces will change. Also, we've been gradually changing the parser due to security issues, having introduced sqlglot as a dependency in https://github.com/apache/superset/pull/26476. Because of this, I thought it wouldn't be necessary to write a SIP. I'm happy to write one if people think it's necessary.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

N/A
TESTING INSTRUCTIONS

Updated unit tests, with mostly cosmetic changes because the SQL pretty-printing has changed.
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

I think this is a welcome change and a move in the right direction, but I feel a big change like this should probably be discussed in the form of a SIP first. I anticipate the final implementation to end up looking similar to this PR, but it may be beneficial to discuss the change first before jumping directly into the code.

I think this is a welcome change and a move in the right direction, but I feel a big change like this should probably be discussed in the form of a SIP first. I anticipate the final implementation to end up looking similar to this PR, but it may be beneficial to discuss the change first before jumping directly into the code.

SIP here: https://github.com/apache/superset/issues/26786

Thanks @betodealmeida !

https://github.com/apache/superset/issues/26786 approved!

Where are we at with this work? There's some additional changes that I'd be willing to take on (https://github.com/apache/superset/pull/19572) but looks like this is a dependency.
We've made and validated significant performance improvements in our Superset fork, but makes sense to consolidate those into SQLScript and SQLStatement rather than directly replace sqlparse calls with sqlglot.

Hey @betodealmeida , this PR might have broken a Hive-specific test -> https://github.com/apache/superset/actions/runs/8272398041/job/22634142747#step:11:608
It appears to fail after this PR was merged (I noticed since it was failing on some other unrelated PR) and trace it back here. Note that this particular GHA test for Hive runs only when superset/** is altered by a PR, that explains why other builds succeeded on master since then.
Hive seems to be complaining about a GROUP BY 1 but from memory it totally supports that. Unclear what's up with the test but I thought you might know more given this PR.


More flexibility with multitouch gesture events (Advances, Optimization, ConconJS)

Sorry for this new PR, but now it's very better.

Add the global position on all changedTouches object
Best way to gesture the double tap event on touchstart and getting each position.
add optimisation and speed up the code
add compatibility support for CocoonJS
add more optimization

We do not accept PRs to master. If this is a change you want to make to v2, then please do it in the v2.x maintenance branch. dev/master will both be v3 soon.


Update example python view code to avoid KeyErrors

If the reader doesn't happen to have the 'date' key in one of the documents the old code would cause couchdb to throw. If the reader happened to be using futon, it would produce a useless 'no response' error.
By using .get() you'll always at least get None and this code can be used against any database.

More correct and pythonic version:
def fun(doc):
    if 'date' in doc:
        yield doc['date'], doc

@iandanforth thanks for that, merged!
@kxepal I don't really agree that your version is significantly more Pythonic. Also, I don't think it's more correct given the fact that there are no falsy values that make up a valid date, anyway.

@djc
If you need to check key in dict, using contains is better than using .get() method. By many reasons, including performance (2.5x times).
Checking here for falsy dates is exactly makes no sense: date value by itself cannot be false, while you remove empty strings and nulls from index. Ok, but in this case you loose ability to find documents with null dates, figure why so and fix that (if needed).


fix(material/chips): don't prevent default mousedown action

The chip row was calling preventDefault on mousedown events in order to avoid sending focus to the chip grid which in turn moves it to the first chip. This is problematic, because preventing the default mousedown action also stops native drag&drop and click events.
These changes resolve to original issue by making the chip focusable on clicks and forwarding focus through there.

I can test this with a screen reader later.


Add unsigned JSON output for Keplr signing

Add a parameter like --generate--only to all instantiate and execute commands that outputs the unsigned JSON that can be copied into a web page for signing with the CosmWasm Stargate signer (which works with Keplr).

Turns out the --generate--only output is specific to the Go implementation of the proto types. It is not the generic proto JSON and this not standardized in any way.
A Go to CosmJS importer is certainly worth doing but it is important to understand there no standard JSON representation.

This could use a few more type checks in the fields but I guess if you copy over the implementation from https://github.com/cosmos/cosmjs/compare/json-importer you can start with this.

The work is available in https://github.com/cosmos/cosmjs/pull/1083. It should cover the use case needed here. For CosmJS we'll probably want to have something more complete before we ship it so I don't see it released soon. Feel free to copy over the implementation. Let me know if it works and how to improve.

All tasks now output JSON strings that can be submitted via /tx.


a few edits

added default diskUsageCommand,
fixed default cpuTemperatureCommand,
added showqr variable. now qrcode can be hide or show.
added Turkish translation.
fixed some css issues.

Hi buddy!
I took a look at your code, and I really like the option to hide the QR — I never thought of that!
I’ve left some comments for you.
A few changes can be made with custom CSS in your Magic Mirror installation, while others are unnecessary.
Take a look when you can!


[BUG] Massive codeblock detection change

Describe the bug
Massive codeblock detection is based of character count
Expected behavior
Detection should be based of line count
Why?
In more accurately describes when codeblock is massive because massive codeblock can be described as "codeblock that takes to much space on the screen" so it matters vertically only
Additional notes
Threshold of around 30-45 lines should be fine

This is a feature request, not a bug.

We cannot base this on line count.
Line lengths are different between clients (web, desktop, mobile), OSs (windows, mac, android) and screen resolutions and scalings.
A limitation on line count would not be any more arbitrary than a limit on character count.

Character count is more often than not a miss. Yes, line count should be adjusted to count for wrapping. We can for example count long lines as 2.


[Feature] Optional Decorators

One sentence description of the feature you would like to request
Add decorators which can optionally be used to set the default value of keys.
Describe the solution you'd like
Probably something like

const builder = new ConfigBuilder<..>()

export const configKey = builder.createDecorator() // creates decorator which are bound to this config to avoid the package having to supply some global variable or the user having to pass in the config they want to fetch from.

class XY{
  @configKey("valid.key")
  readonly configureableKey:string;
}

// decorator should be typed so it won't accept an invalid property type

Actually... I think using this would be a pretty bad idea.
Especially when thinking about testability.
Will close this until there is an request


Improve troubleshooting/docs for gfycat integration

PR for #3225

@justinegeffen looks good, merged. Waiting for Amy's review now. Thanks! :)


Update caching.py

Added 'configured' to redis and mongodb dependencies documentation

Codecov Report

Merging #97 (59ff974) into master (5fe542a) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #97   +/-   ##
=======================================
  Coverage   81.54%   81.54%
=======================================
  Files          26       26
  Lines         634      634
=======================================
  Hits          517      517
  Misses        117      117

Impacted Files
Coverage Δ

pygbif/caching.py
31.57% <ø> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5fe542a...59ff974. Read the comment docs.


Restore Python3 compatibility

execfile was dropped from Python 3, but this is the standard alternative. The current master breaks py3, but I've confirmed that this patch works with Python 3.4.

Thanks @nerdcha  This is a timely fix


Don't perform query on empty input

When a user clicks enter but no query is placed, should ignore the input and not run a search or display the "No results" dropdown.
I made a small edit in my own copy in the "geocode" function of the Geocoders that simply returns if (query.length == 0) that works. I'm not very familiar with using github though or the rest of the code, so I'll just mention it and leave it up to someone else to add to the on-line code if they'd like.

Well, step 1 of being bad at github, this is not the code I meant to comment on. Ignore this.

did you mean to submit here?


Conflict with PHP Intelliphense extension

Not sure if this is fixable on either end specifically, so starting here as its always easier to get a newer project to make a change than a long established one.
So it goes like this.

Load up project with intelliphense.
Scope to a child folder that includes files that call functions/classes from the global scope. ex. a WordPress plugin within the WordPress installation that calls WP core functions.
Reload the window (common for variety of reasons such as extension removed etc).
Notice that intelliphense is now not picking up any of the functions/classes outside of the scoped folder.

Reverse is simple, unscope and "reload window" again, then scope back in.
So just off the top of my head a possible simple solution might be:

Detect window reload, assuming you can. Move the scope to a different storage key.
After reload and init, move the scope settings back into place

This could be cool in general as it could be done for closing the program and opening a project etc.

I don't think it is related directly to the extension logic. This extension just changes the file.exclude setting so VSCode will hide the files in the explorer view. You can try to set it manually (without this extension) and test if it works.
I think it is better to report this issue to intelliphense and explain that their extension doesn't work if you set the file.exclude settings.

@rhalaly Technically speaking their plugin is respecting the excluded files list and not indexing files, functions etc in those paths.
Their extension works fine when you set file.exclude but of course any files excluded don't get indexed or show up for autocomplete etc.
This would likely be true for any extension that ran scans on the project.
The issue is that this scope extension is manipulating that setting for a specific gain, but since other extensions also rely on it I'd think your end likely needs to figure out how to do what it does with as little side effect as possible.

The issue is that this scope extension is manipulating that setting for a specific gain, but since other extensions also rely on it I'd think your end likely needs to figure out how to do what it does with as little side effect as possible.

There are a lot of extensions that manipulate the file.excludes for hiding files for the user, like the extension that I took inspiration from. And personally, I used a lot of them for scoping before I wrote this extension.
Does everyone need to change how they manipulate the file.exclude setting because they can conflict with intelephense?
It is not so rare to have a conflict between extensions. But I (and other similar extensions) can't handle all the conflicts with all other extensions.
The alternative you suggested can't work, AFAIK. The VSCode determines the extensions' initialization order (based on their priority). So, you can't rely on that their index will always run before this extension.
I think the best solution to fix this conflict is to fork this extension and manipulate it for your needs. For example, the scope feature will write to file.exclude as usual, but also write to ignore it in intelephense.files.exclude, so their index will not use it.
The forking doesn't take too much effort. You can build your private extension locally even without publish it if you don't want to.

@rhalaly - I use that exclusion list regularly too, but for its mainly intended purpose of excluding files to lower memory consumption and improve performance, such as node_modules & vendors folders etc.
I understand that is likely the only feasible way you can accomplish this given the limitations of VS Code cores exposed APIs.

The solution you proposed won't do what I'm after, at least not as proposed, I'm looking for intelliphense to index those files, thus they should not be excluded when that happens.
It does appear that the indexer middleware kicks off pretty much immediately so load order is as you pointed out an issue here.
That said I did did into their code and found that they register a command to force reindexing.
I'm wondering if a possible solution might be to add an option for this extension that if set runs a small loop that could be used to counter this on a per user basis, without complexities of making it work with each of them yourself.
Trust me I understand the balance of interacting inside a huge open source plugin/extension based ecosystem. I created and run a top 100 WordPress plugin with over 600k users that integrates very tightly with dozens of other 3rd party WP plugins. (Popup Maker for reference).
What I'm wondering if you'd consider might look like this (pseudo code) during your activation process, and may need some tweaking obviously to get just right, but I haven't studied your plugin's code directly just yet.
if ( preActivation.length ) {
    // 1. Disable current scope and set it into a variable.
    // 2. For each preActivation VSC Command or function, check if it exists and call it.
    // 3. Reactivate current scope from earlier.
}

In this way we could register a custom setting as an array of [CMD]. Not sure functions are feasible as I haven't really studied how much interaction is possible between extensions without a fully exposed CMD, so it may just be CMDs.
But in this way at least there is feasibly a way to get around it when it is an issue.
Just to be clear, I see this issue several times a day, just one of those annoying things I'm trying to live with until a solution is worked out.
Ideally there would be another way for you to manage the scope rather than file.exclude but it is what it is for now until VSC core changes I imagine.

I'm closing this issue as I don't have any alternative for implementing this logic. PR are welcom to assist with this issue


Role associated with lamba doesn't have permission to SQS queue

Using this plugin (Awesome, love it!) and I can't seem to get the role configuration to work properly.
So here's an extract from my serverless.yml
service: testProject

provider:
  name: aws
  runtime: dotnetcore1.0

plugins:
  - serverless-plugin-lambda-dead-letter

functions:
  Handler-TestHandler:
    handler: CsharpHandlers::Test.testProject.Handler::TestHandler
    deadLetter:
      targetArn:
        GetResourceArn: TestDlq
    events:
      - stream: arn:aws:kinesis:ap-southeast-2:*:stream/TestStream

resources:
  Resources:
    TestStream:
      Type: AWS::Kinesis::Stream
      Properties:
        Name: TestStream
        ShardCount: 1
    TestDlq:
        Type: AWS::SQS::Queue
        Properties:
          QueueName: TestDlq
    TestDlqPolicy:
        Type: AWS::SQS::QueuePolicy
        Properties:
          Queues:
            - Ref: TestDlq
          PolicyDocument: {
            "Version":"2012-10-17",
            "Statement":[
              {
                "Effect":"Allow",
 	              "Principal": "*",
                "Action":["SQS:*"],
                "Resource":"*"
              }
            ]}

The SQS is getting created with these permissions (I've really opened it up just to get it to work, then i'll scale it back correctly once i accomplish that)

(Im not even sure if I need to give the above policy document, I would have thought permission would come from the functions ROLE, much like how the Kinesis stream permissions work, through the role)
The SQS is being associated correctly to the lambda function:

However as you can see I have a warning that the Role does not have permissions to that SQS queue
I've got a role being created with the following inline policy which can correctly access the Kinesis stream I have asked it subscribe to, and to CloudWatch in general...
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream"
            ],
            "Resource": "arn:aws:logs:ap-southeast-2:*:*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:ap-southeast-2:*:*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "kinesis:GetRecords",
                "kinesis:GetShardIterator",
                "kinesis:DescribeStream",
                "kinesis:ListStreams"
            ],
            "Resource": [
                "arn:aws:kinesis:ap-southeast-2:*:stream/TestStream",
            ],
            "Effect": "Allow"
        }
    ]
}

I would have assumed that the above general policy would also specific access to the SQS DLQ, but it does not?

Hi AirEssy,
I had a few iterations trying to figure out the policy as well.  With the proposal from #10 there might be a more simplified syntax that would compile the queue resource and policy document for you.
Until then here is the syntax I used in the custom resources...
I can't say which one of these is causing your issue but I noticed a few differences:

You seem to missing the ID in the policy document.
The Statement in the policy document might need an SID as well.
For Principal I used AWS: "*"
For Resource I directly referenced the queue.
I added a condition on the arn so the policy only applies when it matches the Lambda Function arn.

To reference the lambda functions in the Fn::GetAtt you have to infer what serverless is going to use as the Lambda logical id. In CF.  In my case serverlessLambda ==> ServerlessLambdaLambaFunction

Let me know if this works for you.
functions:
  serverlessLambda:
    name: my-serverless-lambda
    description: Lambda to test the serverless utility

    deadLetter:

      # Use the ARN of a resource defined in the resources section.
      targetArn:
        GetResourceArn: ServerlessTplLambdaDlQueue

resources:
    Resources:
      ServerlessTplLambdaDlQueue:
        Type: AWS::SQS::Queue
        Properties:
          QueueName: my-serverless-lambda-retry-queue

      ServerlessTplLambdaDlQueuePolicy:
        Type: AWS::SQS::QueuePolicy
        Properties:
          Queues:
            - Ref: ServerlessTplLambdaDlQueue
          PolicyDocument:

            # This probably needs something more unique
            # like 'Fn::GetAtt: [ServerlessTplLambdaDlQueue, Arn]'  + '/SQSPolicy'
            Id: ServerlessTplLambdaDlQueueId

            Version: '2012-10-17'
            Statement:
              - Sid: Allow-Lambda-SendMessage
                Effect: Allow
                Principal:
                   AWS: "*"
                Action:
                  - SQS:SendMessage
                Resource:
                  Ref: ServerlessTplLambdaDlQueue
                Condition:
                  ArnEquals:
                    aws:SourceArn:
                      Fn::GetAtt: [ServerlessLambdaLambdaFunction, Arn]

Thanks for your reply gmetzker, I've deployed again with

A (very similar) ID on the policy document
Adding the same SID to the statement as above
I've used the same Principal and Action as above
The resource for the policy is ok
I've also followed the pattern above for the ArnEquals, and I can see that has worked as in the SQS permissions, the correct arn is in place.

However the ROLE policy attached to the lambda function is still exactly the same as i described above (I'm not sure if I'm meant to see any reflection of this in there). And i still see the same message under the functions SQS queue selector

I've been able to attach an AWS managed policy to the function's role manually as a work around. We can continue to utilise your extension with that in place, as re-deploying the function doesn't UNDO that policy which is nice. It would be nice however to have our serverless.yml be able to deploy all of our infrastructure correctly for this function without requiring any manual intervention/edits.
Let me know if there's any more information that would be useful to work out how we can accomplish this. Thanks for your quick response/help so far!

Iteresting.  I was under the impression all that was needed was a QueuePolicy, but perhaps the lambda role needs general SQS permission.
In my I'm using an existing IAM role that probably has something like SQS:*
In my environment I typically don't have permissions to modify policies or create roles.  I don't think the plugin should modify existing policies but perhaps it could could compile them in auto-magically if iamRoleStatements were being assigned.
Have you tried setting  the iamRoleStatements described here?  This might be the preferred method.
I need to research a little bit and see if there are analogous scenarios.  Do you know if serverless adds SNS or S3 access to the iamRoleStatements if you specify an SNS or S3 event handler?

@AirEssY I've release version 1.2.0 the syntax should be simplified  and it will eliminate the need to create the queue or policy in the Resources section.
Your Lambda will still need a role that has SQS permissions though.

On second thought i need to do some more investigation on this.  According to the docs serverless can create a default IAM role with rights to some resources.
In my environment I am required to use a predefined role.  So I didn't see that serverless has the ability of creating a role for you.

IMHO, you like won't be creating roles in an enterprise environment. A devops will provide that to you, as in my case. The strange thing is that I had to put this permission on the role rather than the user that is running it. I suppose that is because the Lambda is what needs the SQS perms to publish to it out of EC2, so you need to add AmazonSQSFullAccess as well as AWSLambdaVPCAccessExecutionRole to the role you define your serverless.yml file.


[CI] End rollouts to old cluster

Why
All traffic has been switched to the services running in the general-central cluster.

I can't comment on non-edited code, but there are references to 🚀 Deploy x in old cluster (this for example)
Might be good to rename that to just 🚀 Deploy x 😄  Other than that, LGTM!


How to modify composite resource connection details from kcl function?

I am developing a composite using KCL function code to create resources needed for an Azure workload identity, namely a UserAssignedIdentity, a FederatedIdentityCredential and a K8 Object for a service account.  The PrincipalId of the UserAssignedIdentity is needed by other composites to facilitate the creation of RoleAssignment resources for their own access needs.  So I need to surface the PrincipalId property from the composite somehow.  NOTE: RoleAssignment only accepts a PrincipalId and has no selector option.
The problem is UserAssignedIdentity does not surface any information in connection details so writeConnectionSecretToRef produces a secret with no data, and of course this then provides no data to propagate up to the composite itself.
The status.atProvider data surfaced by the UserAssignedIdentity in the observed ocds dictionary does contain all the data needed of course but I'm not sure how to surface this to the composite.  Theoretically I could write my own secret with the required data but was wondering if there was a way of still using a composite's connectiondetails so the secret is written by crossplane. Any ideas?

If I have to go down the road of creating my own secret, does KCL support transforming some KCL data object to json and base64 encoding it?

Could the system modules help？ https://www.kcl-lang.io/docs/reference/model/json

For format conversion yes it would.  Still need to base64 encode too.  Is there a module with that support available?

Got it https://www.kcl-lang.io/docs/reference/model/base64

@Peefy Whilst I can create a secret from the composition I'm not sure it is the right approach for compositions.
Have you seen how the go templating function supports CompositeConnectionDetails?  The readme explains how to render out a resource to convey composite key/values and also how to indicate if a resource is ready or not (another feature I think is quite important).
That function adds the connection details as defined by their custom resource as shown here.  Ready state detection is by annotation as shown here.  Are both these features something KCL function could implement?

Make sense. This feature can be implemented.

Ready state detection is by annotation as shown here

Hi @Peefy .  Just wanted to be sure we don't forget about ready state annotation enhancement.  Maybe a separate PR?
An example of why it is needed:
If I am copying data from resource A's status.atProvider property into resource B's forProvider property and the resource A is not ready so no data available, I want to still create resource B's MR but make it NotReady (by applying the annotation conditionally).  This assumes resource B can exist synced with the property not yet having a real value.
Only option at the moment is to conditionally not include resource B in the items list.  On the other hand, always including it even though it is not ready runs the risk of it not getting resolved with the correct values.

Just wanted to be sure we don't forget about ready state annotation enhancement. Maybe a separate PR?

I did it in this PR https://github.com/crossplane-contrib/function-kcl/pull/61 and the example:https://github.com/crossplane-contrib/function-kcl/pull/61/files#diff-5d9851a7828526bc8ae1407fef1440e7cdf703f59da4f3ef6a6af59e95ac44d5 because they are related features

I see the annotation code in the recent commit  Thank you.👍
Did you say you had an example of applying the annotation in your linked example?

Yes, do you mean this example with the ready annotation?

I must be blind.  Sorry for not seeing that.

Hey just thinking, is the resource defaulted to ready: True if no annotation is provided?

I see the default ready status is Unspecified. For convenience, it may be possible to set the default value of ready to True from the configuration of kcl-funding itself. For example
      apiVersion: krm.kcl.dev/v1alpha1
      kind: KCLRun
      metadata:
        annotations:
          "krm.kcl.dev/default_ready": "True"
        name: basic
      spec:

Unspecified would make sense as a default I guess.  I'd assume the resource provider itself would then set ready status as and when it needs to.  Setting the annotation to False basically prevents the provider from making a change.
I shall be able to confirm such behaviour very early on with all the compositions I'm creating.

I think too.

You've been very busy and I greatly appreciate all your efforts.  Thank you.

It's okay, I'm willing to communicate anytime. I believe there will be more community participants in the future. ❤

I've tried connection details and works perfect.  I get my resource data propagating up to the secret specified by my claim.
Also tested setting status property on dxr and that works great too.  Haven't tested the ready state yet.

I'm pretty sure the annotation is removed by the meta.RemoveAnnotations call which is fine.  Can't explain why the ready state is not False though.

I think my problem has more to do with the Object in kubernetes-provider and how its ready state is not determined based on the secret manifest state.  I'll try and few things and report back over the weekend.

@Peefy I've learnt a little more on ready state and it's become clear to me that setting it from the function is a use case only necessary for when a resource in a composition has no ready state property that is managed by itself, e.g. a ProviderConfig kind from provider-sql or provider-kubernetes.
For these apis we need to force the ready state else the composition will never become ready, hence need to use the annotation.
I've a real-world use case using the config from provider-sql and it works perfectly fine there.

Hello @markphillips100 Sorry for the late reply. We just had a short vacation.
Are you saying that you need to force the ready state to be set to False in the KCL function? I tried a use case locally and it can indeed be set to False through comments.

Sorry to interrupt the hols.
I was thinking I could set to False in an attempt to allow me to decide the ready state of a resource.  I was trying it with provider-kubernetes Object api to manage the creation of a secret, i.e. make it not ready until the data of the secret had a proper value (one provided by another resources atProvider data).
Seems the Object's ready state is controlled by the provider/crossplane and is made ready by itself when the resource is created, thereby ignoring any value the function may have set.  You can attempt to use the Object's readiness policy DeriveFromObject but I believe in this case it makes no difference.

I tried a use case locally and it can indeed be set to False through comments.

Curious what you mean by "through comments"?

Make sense. I'll give a try.

Closed by #61


Add support for registering event handlers with shared_ptr and weak_ptr

Fixes #1316

The implementation seems reasonable, but I'll let @dmachaj sign off.

Do either of you know if these delegates are pruned by the linker if there is no usage?  The only meta-question I have is regarding binary size.  If these delegates have two additional vtable slots each, and they are not pruned by the linker when unused, then that might add up across a large code base.

@sylveon thank you for taking this on 🙂.  I am now back from my vacation and it was a nice thing to see this PR at the top of my inbox.  I think your changes here (especially the test coverage) are better than anything I would have been able to write 👍.

Since all the implementation is templated constructors, I expect them to never even show up to the linker until used (because the compiler wouldn't instantiate the template for no reason).
And you're welcome @dmachaj! I had some free time this weekend so I decided why not do this :)

If these delegates have two additional vtable slots each

This does not affect binary (vtable) layout.


ArcFormats: Support S5IC archieve

Tested game: https://vndb.org/r119690

Add UTF-16 support for BinaryStream::ReadCString(int, Encoding) method

I don't think it's necessary to modify ReadCString here. A better solution would be to add a ReadUnicodeString method.
string ReadUnicodeString(int length, bool isNullTerminated = true)

I don't think it's necessary to modify ReadCString here. A better solution would be to add a ReadUnicodeString method.
string ReadUnicodeString(int length, bool isNullTerminated = true)

You mean implement the method for this format only or add it to IBinaryStream interface and implement it on all subclass?

Sorry, I found that FindEoS is already implemented in BinMemoryStream, and it's really better to modify ReadCString for good consistency. Thank you for your contribution.


HELP:UserWarning: None of the inputs have requires_grad=True. Gradients will be None

import network module: networks.lora
create LoRA network. base dim (rank): 4, alpha: 1
create LoRA for Text Encoder: 72 modules.
create LoRA for U-Net: 192 modules.
enable LoRA for text encoder
enable LoRA for U-Net
prepare optimizer, data loader etc.
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
For effortless bug reporting copy-paste your error into this form: https://docs.google.com/forms/d/e/1FAIpQLScPB8emS3Thkp66nvqwmjTEgxp8Y9ufuWTzFyr9kJ5AoI47dQ/viewform?usp=sf_link
/root/miniconda3/envs/cxk/lib/python3.9/site-packages/bitsandbytes/cuda_setup/paths.py:93: UserWarning: /root/miniconda3/envs/cxk did not contain libcudart.so as expected! Searching further paths...
warn(
/root/miniconda3/envs/cxk/lib/python3.9/site-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/nvidia/lib64'), PosixPath('/root/miniconda3/envs/cxk/lib/python3.9/site-packages/cv2/../../lib64'), PosixPath('/usr/local/nvidia/lib')}
warn(
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 113
CUDA SETUP: Loading binary /root/miniconda3/envs/cxk/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda113.so...
use 8-bit AdamW optimizer | {}
running training / 学習開始
num train images * repeats / 学習画像の数×繰り返し回数: 1710
num reg images / 正則化画像の数: 0
num batches per epoch / 1epochのバッチ数: 1710
num epochs / epoch数: 1
batch size per device / バッチサイズ: 1
gradient accumulation steps / 勾配を合計するステップ数 = 1
total optimization steps / 学習ステップ数: 400
steps:   0%|                                                                                | 0/400 [00:00<?, ?it/s]epoch 1/1
/root/miniconda3/envs/cxk/lib/python3.9/site-packages/torch/utils/checkpoint.py:25: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
warnings.warn("None of the inputs have requires_grad=True. Gradients will be None")

I believe this is because you have gradient_checkpoints set to true. And if so, it will do this check. Problem is that I don't know if gradient accumulation steps (GAS) are actually supported for LoRA training, as most of the examples I've seen all have it disabled and set to 1. If anyone knows if GAS actually works despite of this error (which I'm assuming gradients don't work), let me know.

ありがとうございます！

The gradient checkpointing seems to work actually for the LoRA training. I've fixed in this commit: e6a8c9d
Grandient checkpointing seems to be enabled by specifying requires_grad_(True) for the first parameter of the model. However, LoRA training does not train that parameter, so this warning seems to appear.

ありがとうございます！

I believe this is because you have gradient_checkpoints set to true. And if so, it will do this check. Problem is that I don't know if gradient accumulation steps (GAS) are actually supported for LoRA training, as most of the examples I've seen all have it disabled and set to 1. If anyone knows if GAS actually works despite of this error (which I'm assuming gradients don't work), let me know.

thanks！

The gradient checkpointing seems to work actually for the LoRA training. I've fixed in this commit: e6a8c9d
Grandient checkpointing seems to be enabled by specifying requires_grad_(True) for the first parameter of the model. However, LoRA training does not train that parameter, so this warning seems to appear.

So I can ignore it while training a LoRa?

No the problem is not solved in the latest updated of Kohya_SS
use AdamW optimizer | {}
running training / 学習開始
num train images * repeats / 学習画像の数×繰り返し回数: 6720
num reg images / 正則化画像の数: 0
num batches per epoch / 1epochのバッチ数: 1680
num epochs / epoch数: 16
batch size per device / バッチサイズ: 4
gradient accumulation steps / 勾配を合計するステップ数 = 1
total optimization steps / 学習ステップ数: 26880
steps:   0%|                                                                                | 0/26880 [00:00<?, ?it/s]
epoch 1/16
E:\kohya_ss\venv\lib\site-packages\torch\utils\checkpoint.py:31: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
warnings.warn("None of the inputs have requires_grad=True. Gradients will be None")

This problem also occurs when I use sampling an image every other epoch.

Hi I came here to check about this warning, so okay gradient checkpointing (whatever this is) is not used by Lora training, but.... if I don't check that box the training would be easily 10 times slower. Very strange... oh well, as long as it works I guess ? shrugs


[EPIC] Rollups fraud proofs using QGB

As part of the efforts dedicated towards having an example of a Celestium, we will need to implement the following:

[ ] https://github.com/celestiaorg/quantum-gravity-bridge/issues/141
[ ] https://github.com/celestiaorg/protobuf3-solidity/issues/19
[ ] https://github.com/celestiaorg/quantum-gravity-bridge/issues/133

Hmmm. What's the need for decoding the PFB transaction itself? The point of posting the data root was so that the blobs can be used without looking at transactions. An alternative would be to revert to using (potentially in addition to the data root) tuple roots and have each tuple include addition metadata.

Thats to prove that a transaction, which has a specific commitment, was posted to Celestia.
We could use the blobs directly, but the proofs will be significantly bigger since we will need to send the whole rollup block to the solidity smart contract to be verified, as opposed to sending only the PFB.
Also, using the blobs or the subtree root doesn't allow us to check whether the blobs were posted to the provided shareIndex. Since, we can prove that some data was posted on some block, but we can't prove its position.

I'm trying to find where the current scheme was first proposed but I'm drawing a blank. IIRC @musalbas suggested it.
Basically, in the current scheme rollup block producers post a new block to Ethereum and claim it uses a certain range or ranges of shares. The fun thing is those shares don't actually have to be a complete blob, or even from a single blob!
The fraud proof would then prove that a single tx in the rollup is 1) in the data root (only requires proving the tx, not the whole block), and 2) is in the range(s) claimed (trivial).

@adlerjohn

in the data root

We can prove that a transaction was posted in the block, but how would we prove that that transaction is the right transaction? We will need to parse it to verify if the commitment and share index is the same.

is in the range(s) claimed (trivial)

this means proving its position in the block? How would we do that?

how would we prove that that transaction is the right transaction

You don't actually need to prove anything about transactions, or blobs. That's the neat part. The rollup block producer just claims "here's this range in the Celestia block, that's the rollup block." That range could be anything. Doesn't have to be a transaction or a blob.

This means proving its position in the block? How would we do that?

Something along the lines of https://github.com/celestiaorg/nmt/pull/49

I mean even if you wanted to prove the commitment of the blob (without using a range), can't you still do that by using a blob inclusion proof - not a pfb proof?


Bioconductor devel submission

Could we bump the current master to Bioconductor devel? There are some function updates that I would need in examples in a PR to philr package (to be added), and it would be best if the functionality was available through the latest Bioc devel version (rather than from Github master branch).

done


Fix #279

Again ;)
Will this possibly be the last time?

Codecov Report

Merging #574 into master will decrease coverage by 1.07%.
The diff coverage is 94.24%.

@@            Coverage Diff             @@
##           master     #574      +/-   ##
==========================================
- Coverage   85.81%   84.74%   -1.08%
==========================================
  Files          36       36
  Lines        6676     6685       +9
==========================================
- Hits         5729     5665      -64
- Misses        947     1020      +73

Impacted Files
Coverage Δ

gpiozero/pins/native.py
36.53% <0%> (-27.89%)
:arrow_down:

gpiozero/pins/rpio.py
3.2% <0%> (ø)
:arrow_up:

gpiozero/pins/pigpio.py
1.38% <0%> (-0.33%)
:arrow_down:

gpiozero/pins/rpigpio.py
3.17% <0%> (ø)
:arrow_up:

gpiozero/input_devices.py
87.5% <100%> (ø)
:arrow_up:

tests/test_boards.py
100% <100%> (ø)
:arrow_up:

gpiozero/pins/mock.py
97.26% <100%> (-0.01%)
:arrow_down:

gpiozero/pins/local.py
93.91% <100%> (-0.61%)
:arrow_down:

tests/test_inputs.py
100% <100%> (ø)
:arrow_up:

gpiozero/output_devices.py
99.47% <100%> (ø)
:arrow_up:

... and 12 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5a07b54...9fdc093. Read the comment docs.


Show source file in source code blocks titles

Instead of just "Show source code", we should write "Show source code in mypackage/mymodule.py".
We will need to find a way to compute the relative path of the files.

Implemented here: https://github.com/pawamoy/mkdocstrings/commit/f41c11988d8d849a0310cca511c2d93a74cab86f#diff-2795c5aa918a9b137bdd26a39259c13bR156


Bumped Mesos SHA for dc/os 1.11.5 container cleanup EBUSY fix.

High-level description
What features does this change enable? What bugs does this change fix?
Corresponding DC/OS tickets (obligatory)
These DC/OS JIRA ticket(s) must be updated (ideally closed) in the moment this PR lands:

DCOS-41224 Removing rootfs mounts may fail with EBUSY.

Related tickets (optional)
Other tickets related to this change:

MESOS-9196 Removing rootfs mounts may fail with EBUSY.

Checklist for all PRs

[ ] Added a comprehensible changelog entry to CHANGES.md or explain why this is not a user-facing change:
[x] Included a test which will fail if code is reverted but test is not. If there is no test please explain here:
[x] Read the DC/OS contributing guidelines
[x] Followed relevant code rules Rules for Packages and Systemd

Checklist for component/package updates:
If you are changing components or packages in DC/OS (e.g. you are bumping the sha or ref of anything underneath packages), then in addition to the above please also include:

[x] Change log: here
[x] Test Results: here
[ ] Code Coverage (if available): [link to code coverage report]

@mesosphere-mergebot bump-ee

@mesosphere-mergebot label Ready For Review

@mesosphere-mergebot merge-it

This should have gotten a CHANGELOG entry.
CC @jieyu @rukletsov @karya0


Remove all curses/blessings from the modifiers deck when starting a new game

A new game is started when the user clicks on the 'New Game' button in the Play tab

This pr should fix this issue https://github.com/orichalque/yet-an-other-gloomhaven-companion/pull/128

This pr should fix this issue https://github.com/orichalque/yet-an-other-gloomhaven-companion/pull/128


Unhandled strided slice case

I'm having an issue with a strided slice op in a model i'm working on.
This is the stack trace:
Traceback (most recent call last):
  File "convert.py", line 40, in <module>
    image_input_names=image_input_name)
  File "/home/baris/.virtualenvs/deepfigures/lib/python3.5/site-packages/tfcoreml/_tf_coreml_converter.py", line 586, in convert
    custom_conversion_functions=custom_conversion_functions)
  File "/home/baris/.virtualenvs/deepfigures/lib/python3.5/site-packages/tfcoreml/_tf_coreml_converter.py", line 337, in _convert_pb_to_mlmodel
    convert_ops_to_layers(context)
  File "/home/baris/.virtualenvs/deepfigures/lib/python3.5/site-packages/tfcoreml/_ops_to_layers.py", line 178, in convert_ops_to_layers
    translator(op, context)
  File "/home/baris/.virtualenvs/deepfigures/lib/python3.5/site-packages/tfcoreml/_layers.py", line 1269, in strided_slice
    assert False, ('Strided Slice case not handled. Input shape = {}, output shape = {}'.format(str(input_shape),str(output_shape)))
AssertionError: Strided Slice case not handled. Input shape = [1, 20, 15, 832], output shape = [1, 20, 15, 132]

I've printed out some related local variables to help pinpoint what the issue is:
input_shape:  [1, 20, 15, 832]
begin:  [  0   0   0 700]
strides:  [1 1 1 1]
end:  [0 0 0 0]
begin_mask:  7
end_mask:  15

Would it be possible to support strided slices of this form?

Same here trying to export DeepSpeech model as CoreML:
AssertionError: Strided Slice case not handled. Input shape = [16, 1, 2048], output shape = [1, 2048]

@suoloordi @lissyx Even I am having a similar problem.
AssertionError: Strided Slice case not handled. Input shape = [1, 4, 26], output shape = [1, 3, 26]
Did you find any workaround for this problem?

Nope

@lissyx For DeepSpeech model, it's possible to get rid of stripped slice op by using strip_unused_lib.strip_unused utility.
from tensorflow.python.tools import strip_unused_lib
from tensorflow.python.framework import dtypes
from tensorflow.python.platform import gfile

tf_model_path = 'output_graph.pb'
with open(tf_model_path, 'rb') as f:
    serialized = f.read()
tf.reset_default_graph()
original_gdef = tf.GraphDef()
original_gdef.ParseFromString(serialized)

input_node_names = []
output_node_names = ['Softmax']
gdef = strip_unused_lib.strip_unused(
        input_graph_def = original_gdef,
        input_node_names = input_node_names,
        output_node_names = output_node_names,
        placeholder_type_enum = dtypes.float32.as_datatype_enum)
# Save it to an output file
frozen_model_file = './striped_frozen.pb'
with gfile.GFile(frozen_model_file, "wb") as f:
    f.write(gdef.SerializeToString()) ```

@lissyx For DeepSpeech model, it's possible to get rid of strided slice op by using strip_unused_lib.strip_unused utility.
from tensorflow.python.tools import strip_unused_lib
from tensorflow.python.framework import dtypes
from tensorflow.python.platform import gfile

tf_model_path = 'output_graph.pb'
with open(tf_model_path, 'rb') as f:
    serialized = f.read()
tf.reset_default_graph()
original_gdef = tf.GraphDef()
original_gdef.ParseFromString(serialized)

input_node_names = []
output_node_names = ['Softmax']
gdef = strip_unused_lib.strip_unused(
        input_graph_def = original_gdef,
        input_node_names = input_node_names,
        output_node_names = output_node_names,
        placeholder_type_enum = dtypes.float32.as_datatype_enum)
# Save it to an output file
frozen_model_file = './striped_frozen.pb'
with gfile.GFile(frozen_model_file, "wb") as f:
    f.write(gdef.SerializeToString()) ```

Interesting, last time I tried strip_unused it was not really doing any good. I'll give it a new try then!

@DawerG As before, applying strip_unused, I still have the same list of ops in the output graph, including StridedSlice.

@lissyx For deepspeech model, please refer to https://github.com/tf-coreml/tf-coreml/blob/master/examples/deep_speech.ipynb

@lissyx For deepspeech model, please refer to https://github.com/tf-coreml/tf-coreml/blob/master/examples/deep_speech.ipynb

Can you be more specific ? This is 0.4.1. Was the strided slice issue a bug on tf-coreml side that was fixed ?

@suoloordi This particular example of strider-slice works fine. Can you please try building from the source?


kuksa_client: Implement new gRPC/proto API

This PR allows kuksa-client to communicate with kuksa-databroker using latest gRPC/proto API (kuksa.val.v1).

So I tried to do some tests following the README file.
Starting the docker file seems to fail, I assume logging.ini isn't included and needs to mentioned in setup.cfg
erik@debian3:~/kuksa.val/kuksa-client$ docker run --rm -it --net=host kuksa-client:latest
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/kuksa-client/kuksa_client/__main__.py", line 425, in <module>
    sys.exit(main())
  File "/kuksa-client/kuksa_client/__main__.py", line 420, in main
    logging.config.fileConfig(args.logging_config)
  File "/usr/local/lib/python3.10/logging/config.py", line 72, in fileConfig
    formatters = _create_formatters(cp)
  File "/usr/local/lib/python3.10/logging/config.py", line 105, in _create_formatters
    flist = cp["formatters"]["keys"]
  File "/usr/local/lib/python3.10/configparser.py", line 964, in __getitem__
    raise KeyError(key)
KeyError: 'formatters'

I also tried to build locally, but there the instructions seems to be outdated:
erik@debian3:~/kuksa.val/kuksa-client$ pipenv shell
Launching subshell in virtual environment...
 . /home/erik/.local/share/virtualenvs/kuksa-client-i_dVn3q4/bin/activate
erik@debian3:~/kuksa.val/kuksa-client$  . /home/erik/.local/share/virtualenvs/kuksa-client-i_dVn3q4/bin/activate
(kuksa-client) erik@debian3:~/kuksa.val/kuksa-client$ python __main__.py
python: can't open file '/home/erik/kuksa.val/kuksa-client/__main__.py': [Errno 2] No such file or directory
(kuksa-client) erik@debian3:~/kuksa.val/kuksa-client$

(This is I believe not related to this PR, but should better be fixed anyway. For both local build and docker run it could maybe be good in the README to show how to select grpc or viss when starting the client)

@erikbosch I have normally addressed your different comments.
Things should work better now and should also be better documented. ;)


test(ts): Added a TS compilation check

Platforms affected
Motivation and Context

Adds a TS check on our type def files.
Description

Adds Typescript 5.1 as a dev dependency and updates the test script to run tsc to see if our types will at least compile.
There is an empty .ts file as an entry point for tsc, but it could be expanded in the future to actually make use of the API and ensure that types will compile as expected.
Testing

Ran npm test
Tests will fail without a PR fix such as https://github.com/apache/cordova-plugin-file/pull/588
Checklist

[x] I've run the tests to see all new and existing tests pass
[x] I added automated test coverage as appropriate for this change
[x] Commit is prefixed with (platform) if this change only applies to one platform (e.g. (android))
[x] If this Pull Request resolves an issue, I linked to the issue in the text above (and used the correct keyword to close issues using keywords)
[x] I've updated the documentation if necessary

This all makes sense to me! 🥇


The type System.Collections.Generic.IEnumerable<StarWars.Models.ICharacter> of field Query.character could not be resolved to a valid schema type

Describe the bug
Starwars template project throws the following error, for any query:
The type `System.Collections.Generic.IEnumerable<StarWars.Models.ICharacter>` of field `Query.character` could not be resolved to a valid schema type
To Reproduce
Steps to reproduce the behavior:

dotnet new starwars
dotnet run --project StarWars/StarWars.csproj -c release
Open GraphiQL and execute any query on http://localhost:5000

Expected behavior
The query executes correctly and returns a result
Desktop:

OS: Windows 10
dotnet core 2.1

I will have a look at it.

hi ThunderDev1,
I can confirm the bug. I will fix it and redeploy the template.

Ok, the issue is fixed and release with the template version 0.4.6.
Release Info
It will take a few minutes for nuget to update its package cache.
Nuget
You can update your nuget template like this:
dotnet new -u HotChocolate.Templates.StarWars
dotnet new -i HotChocolate.Templates.StarWars

This will uninstall the old version and install the new.
Thank you for reporting this issue.  👍


auto_lightgbm not working

Describe the bug
auto_lightgbm is not working, when following the documentation: https://docs.functime.ai/forecasting/#quantile-regression
To Reproduce
import json
from timeit import default_timer

import polars as pl

from functime.cross_validation import train_test_split
from functime.forecasting import auto_linear_model, linear_model, naive, snaive, auto_lightgbm
from functime.metrics import smape
from functime.preprocessing import scale
from functime.seasonality import add_fourier_terms

start_time = default_timer()

y = pl.read_parquet(
    "https://github.com/functime-org/functime/raw/main/data/commodities.parquet"
)
entity_col, time_col = y.columns[:2]
X = y.select([entity_col, time_col]).pipe(add_fourier_terms(sp=12, K=6)).collect()

print("🎯 Target variable (y):\n", y)
print("📉 Exogenous variables (X):\n", X)

test_size = 3
freq = "1mo"
y_train, y_test = train_test_split(test_size)(y)
X_train, X_test = train_test_split(test_size)(X)

y_pred_10 = auto_lightgbm(alpha=0.1, freq="1d")(y=y_train, fh=28)
y_pred_90 = auto_lightgbm(alpha=0.9, freq="1d")(y=y_train, fh=28)

Expected behavior
that the example in the documentation would at least execute
Screenshots

Desktop (please complete the following information):

OS: Mac
Functime version: 0.9.5.

Additional context
As per requested on LinkedIn

Ciao @jarichb, thanks for reporting and sorry for the late reply. I see the error! Did you make sure to install functime[lgb]?
We documented that on this page: how can we improve that?
I think I need to make this error be raised sooner.

interestingly enough, I can't seem to get it installed like that either.
`(.env) jarich@Jarichs-Air functime % pip install "functime[llm,lgb]"
Requirement already satisfied: functime[lgb,llm] in ./.env/lib/python3.8/site-packages (0.9.5)
Requirement already satisfied: cloudpickle in ./.env/lib/python3.8/site-packages (from functime[lgb,llm]) (3.0.0)
Requirement already satisfied: flaml<3,>=2.0.2 in ./.env/lib/python3.8/site-packages (from functime[lgb,llm]) (2.1.2)
Requirement already satisfied: holidays in ./.env/lib/python3.8/site-packages (from functime[lgb,llm]) (0.46)
Requirement already satisfied: numpy in ./.env/lib/python3.8/site-packages (from functime[lgb,llm]) (1.24.4)
Requirement already satisfied: polars>=0.20.7 in ./.env/lib/python3.8/site-packages (from functime[lgb,llm]) (0.20.21)
Requirement already satisfied: scikit-learn<2,>=1.2.2 in ./.env/lib/python3.8/site-packages (from functime[lgb,llm]) (1.3.2)
Requirement already satisfied: scipy in ./.env/lib/python3.8/site-packages (from functime[lgb,llm]) (1.10.1)
Requirement already satisfied: tqdm in ./.env/lib/python3.8/site-packages (from functime[lgb,llm]) (4.66.2)
Requirement already satisfied: typing-extensions in ./.env/lib/python3.8/site-packages (from functime[lgb,llm]) (4.11.0)
Collecting lightgbm (from functime[lgb,llm])
Using cached lightgbm-4.3.0.tar.gz (1.7 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing metadata (pyproject.toml) ... done
Collecting openai (from functime[lgb,llm])
Using cached openai-1.20.0-py3-none-any.whl.metadata (21 kB)
Collecting tabulate (from functime[lgb,llm])
Using cached tabulate-0.9.0-py3-none-any.whl.metadata (34 kB)
Collecting tenacity (from functime[lgb,llm])
Using cached tenacity-8.2.3-py3-none-any.whl.metadata (1.0 kB)
Collecting tiktoken (from functime[lgb,llm])
Using cached tiktoken-0.6.0-cp38-cp38-macosx_11_0_arm64.whl.metadata (6.6 kB)
Requirement already satisfied: joblib>=1.1.1 in ./.env/lib/python3.8/site-packages (from scikit-learn<2,>=1.2.2->functime[lgb,llm]) (1.4.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in ./.env/lib/python3.8/site-packages (from scikit-learn<2,>=1.2.2->functime[lgb,llm]) (3.4.0)
Requirement already satisfied: python-dateutil in ./.env/lib/python3.8/site-packages (from holidays->functime[lgb,llm]) (2.9.0.post0)
Collecting anyio<5,>=3.5.0 (from openai->functime[lgb,llm])
Using cached anyio-4.3.0-py3-none-any.whl.metadata (4.6 kB)
Collecting distro<2,>=1.7.0 (from openai->functime[lgb,llm])
Using cached distro-1.9.0-py3-none-any.whl.metadata (6.8 kB)
Collecting httpx<1,>=0.23.0 (from openai->functime[lgb,llm])
Using cached httpx-0.27.0-py3-none-any.whl.metadata (7.2 kB)
Collecting pydantic<3,>=1.9.0 (from openai->functime[lgb,llm])
Using cached pydantic-2.7.0-py3-none-any.whl.metadata (103 kB)
Collecting sniffio (from openai->functime[lgb,llm])
Using cached sniffio-1.3.1-py3-none-any.whl.metadata (3.9 kB)
Collecting regex>=2022.1.18 (from tiktoken->functime[lgb,llm])
Using cached regex-2024.4.16-cp38-cp38-macosx_11_0_arm64.whl.metadata (40 kB)
Collecting requests>=2.26.0 (from tiktoken->functime[lgb,llm])
Using cached requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting idna>=2.8 (from anyio<5,>=3.5.0->openai->functime[lgb,llm])
Using cached idna-3.7-py3-none-any.whl.metadata (9.9 kB)
Collecting exceptiongroup>=1.0.2 (from anyio<5,>=3.5.0->openai->functime[lgb,llm])
Using cached exceptiongroup-1.2.0-py3-none-any.whl.metadata (6.6 kB)
Collecting certifi (from httpx<1,>=0.23.0->openai->functime[lgb,llm])
Using cached certifi-2024.2.2-py3-none-any.whl.metadata (2.2 kB)
Collecting httpcore==1.* (from httpx<1,>=0.23.0->openai->functime[lgb,llm])
Using cached httpcore-1.0.5-py3-none-any.whl.metadata (20 kB)
Collecting h11<0.15,>=0.13 (from httpcore==1.*->httpx<1,>=0.23.0->openai->functime[lgb,llm])
Using cached h11-0.14.0-py3-none-any.whl.metadata (8.2 kB)
Collecting annotated-types>=0.4.0 (from pydantic<3,>=1.9.0->openai->functime[lgb,llm])
Using cached annotated_types-0.6.0-py3-none-any.whl.metadata (12 kB)
Collecting pydantic-core==2.18.1 (from pydantic<3,>=1.9.0->openai->functime[lgb,llm])
Using cached pydantic_core-2.18.1-cp38-cp38-macosx_11_0_arm64.whl.metadata (6.5 kB)
Collecting charset-normalizer<4,>=2 (from requests>=2.26.0->tiktoken->functime[lgb,llm])
Using cached charset_normalizer-3.3.2-cp38-cp38-macosx_11_0_arm64.whl.metadata (33 kB)
Collecting urllib3<3,>=1.21.1 (from requests>=2.26.0->tiktoken->functime[lgb,llm])
Using cached urllib3-2.2.1-py3-none-any.whl.metadata (6.4 kB)
Requirement already satisfied: six>=1.5 in ./.env/lib/python3.8/site-packages (from python-dateutil->holidays->functime[lgb,llm]) (1.16.0)
Using cached openai-1.20.0-py3-none-any.whl (292 kB)
Using cached tabulate-0.9.0-py3-none-any.whl (35 kB)
Using cached tenacity-8.2.3-py3-none-any.whl (24 kB)
Using cached tiktoken-0.6.0-cp38-cp38-macosx_11_0_arm64.whl (922 kB)
Using cached anyio-4.3.0-py3-none-any.whl (85 kB)
Using cached distro-1.9.0-py3-none-any.whl (20 kB)
Using cached httpx-0.27.0-py3-none-any.whl (75 kB)
Using cached httpcore-1.0.5-py3-none-any.whl (77 kB)
Using cached pydantic-2.7.0-py3-none-any.whl (407 kB)
Using cached pydantic_core-2.18.1-cp38-cp38-macosx_11_0_arm64.whl (1.7 MB)
Using cached regex-2024.4.16-cp38-cp38-macosx_11_0_arm64.whl (291 kB)
Using cached requests-2.31.0-py3-none-any.whl (62 kB)
Using cached sniffio-1.3.1-py3-none-any.whl (10 kB)
Using cached annotated_types-0.6.0-py3-none-any.whl (12 kB)
Using cached certifi-2024.2.2-py3-none-any.whl (163 kB)
Using cached charset_normalizer-3.3.2-cp38-cp38-macosx_11_0_arm64.whl (119 kB)
Using cached exceptiongroup-1.2.0-py3-none-any.whl (16 kB)
Using cached idna-3.7-py3-none-any.whl (66 kB)
Using cached urllib3-2.2.1-py3-none-any.whl (121 kB)
Using cached h11-0.14.0-py3-none-any.whl (58 kB)
Building wheels for collected packages: lightgbm
Building wheel for lightgbm (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building wheel for lightgbm (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [40 lines of output]
2024-04-16 21:15:14,301 - scikit_build_core - INFO - RUN: /private/var/folders/fl/qtsnzd1x0k3dn109zhspr_yc0000gn/T/pip-build-env-rnuvs7bi/normal/lib/python3.8/site-packages/cmake/data/bin/cmake --version
2024-04-16 21:15:14,307 - scikit_build_core - INFO - CMake version: 3.29.2
*** scikit-build-core 0.8.2 using CMake 3.29.2 (wheel)
2024-04-16 21:15:14,312 - scikit_build_core - INFO - Build directory: /private/var/folders/fl/qtsnzd1x0k3dn109zhspr_yc0000gn/T/tmpl1sla9l5/build
*** Configuring CMake...
2024-04-16 21:15:14,322 - scikit_build_core - INFO - RUN: /private/var/folders/fl/qtsnzd1x0k3dn109zhspr_yc0000gn/T/pip-build-env-rnuvs7bi/normal/lib/python3.8/site-packages/ninja/data/bin/ninja --version
2024-04-16 21:15:14,420 - scikit_build_core - INFO - Ninja version: 1.11.1
2024-04-16 21:15:14,422 - scikit_build_core - INFO - RUN: /private/var/folders/fl/qtsnzd1x0k3dn109zhspr_yc0000gn/T/pip-build-env-rnuvs7bi/normal/lib/python3.8/site-packages/cmake/data/bin/cmake -S. -B/var/folders/fl/qtsnzd1x0k3dn109zhspr_yc0000gn/T/tmpl1sla9l5/build -DCMAKE_BUILD_TYPE:STRING=Release -C/var/folders/fl/qtsnzd1x0k3dn109zhspr_yc0000gn/T/tmpl1sla9l5/build/CMakeInit.txt -DCMAKE_MAKE_PROGRAM=/private/var/folders/fl/qtsnzd1x0k3dn109zhspr_yc0000gn/T/pip-build-env-rnuvs7bi/normal/lib/python3.8/site-packages/ninja/data/bin/ninja -D__BUILD_FOR_PYTHON:BOOL=ON
loading initial cache file /var/folders/fl/qtsnzd1x0k3dn109zhspr_yc0000gn/T/tmpl1sla9l5/build/CMakeInit.txt
-- The C compiler identification is AppleClang <IP_ADDRESS>50022
-- The CXX compiler identification is AppleClang <IP_ADDRESS>50022
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES)
-- Could NOT find OpenMP_CXX (missing: OpenMP_CXX_FLAGS OpenMP_CXX_LIB_NAMES)
-- Could NOT find OpenMP (missing: OpenMP_C_FOUND OpenMP_CXX_FOUND)
-- Found OpenMP_C: -Xpreprocessor -fopenmp -I/opt/homebrew/opt/libomp/include
-- Found OpenMP_CXX: -Xpreprocessor -fopenmp -I/opt/homebrew/opt/libomp/include
-- Found OpenMP: TRUE
-- Performing Test MM_PREFETCH
-- Performing Test MM_PREFETCH - Failed
-- Performing Test MM_MALLOC
-- Performing Test MM_MALLOC - Success
-- Using _mm_malloc
-- Configuring done (0.7s)
-- Generating done (0.0s)
-- Build files have been written to: /var/folders/fl/qtsnzd1x0k3dn109zhspr_yc0000gn/T/tmpl1sla9l5/build
*** Building project with Ninja...
2024-04-16 21:15:15,127 - scikit_build_core - INFO - RUN: /private/var/folders/fl/qtsnzd1x0k3dn109zhspr_yc0000gn/T/pip-build-env-rnuvs7bi/normal/lib/python3.8/site-packages/cmake/data/bin/cmake --build /var/folders/fl/qtsnzd1x0k3dn109zhspr_yc0000gn/T/tmpl1sla9l5/build
ninja: error: '/opt/homebrew/opt/libomp/lib/libomp.dylib', needed by '/private/var/folders/fl/qtsnzd1x0k3dn109zhspr_yc0000gn/T/pip-install-ugmjnjqi/lightgbm_3ae15dce17b94645a154c1a1980eb331/lib_lightgbm.so', missing and no known rule to make it
  *** CMake build failed
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for lightgbm
Failed to build lightgbm
ERROR: Could not build wheels for lightgbm, which is required to install pyproject.toml-based projects`

Yes, sounds good to me! Thank you
On Wed, 17 Apr 2024, 17:49 baggiponte, @.***> wrote:

UPDATE: this error was resolved with 'brew install libomp', and is a LGB
specific thing, now I got it to work!
I do think that 'ImportError' not callable is a rather vague error though,
if it could display to make the specific install, I feel like that would
help.
Yes, I should explicitly mark that lightgbm requires libomp. Will add it
to the docs and then proceed to close the issue. Does this sound right to
you? 😊
—
Reply to this email directly, view it on GitHub
https://github.com/functime-org/functime/issues/190#issuecomment-2061617911,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AP4NCDU56APNWAC2XE7XJ5DY52KXBAVCNFSM6AAAAABF5IZGVWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRRGYYTOOJRGE
.
You are receiving this because you were mentioned.Message ID:
@.***>


AIP config is sourced too soon

https://batch.hail.populationgenomics.org.au/batches/429173/jobs/1
Upon importing the Stages from the relevant cpg_workflows.stages files, the AIP config is set
This occurs prior to the setting of config paths (including the workflow default)
Outcome: the fixed config doesn't contain any of the fields required from the default config

Also this https://batch.hail.populationgenomics.org.au/batches/429177/jobs/4


Get KeycloakSecurityContext from session

Fixed #JBPM-6278 NullPointerException when configuring user and group management services using KC adapter provider

Can one of the admins verify this PR? Comment with 'ok to test' to start the build.

Can one of the admins verify this PR? Comment with 'ok to test' to start the build.

Can one of the admins verify this PR? Comment with 'ok to test' to start the build.

Can one of the admins verify this PR? Comment with 'ok to test' to start the build.

Can one of the admins verify this PR? Comment with 'ok to test' to start the build.

Can one of the admins verify this PR? Comment with 'ok to test' to start the build.

Hi @lhwong thanks for this! Could you please add tests for this PR? Also, if this fix JBPM-6278 could you please change your commit message for "[JBPM-6278] NullPointerException when configuring user and group management services using KC adapter provider"


Anoucements Update Feierabend and Global Time

2 new messages:
One global announcing Time-Change.
One LEU announcing Feierabend.

Added technical issues community message

And updated message.


Regarding the recent database dump of Epik's customers

What should I do for myself and my friends who I've recommended this to? I've done the standard changing my password and adding 2fa, but should I be worried about anything else? From my understanding, the dump includes at least full names, addresses, and passwords. Is it worth it to switch to a different domain service?

Change your password, and the password of any service where you were
using the same password (which is always a bad idea). I say this even
though I'm pretty sure Epik has been using a third party authenticator
for the past year or two, so the password hashes leak might be obsolete
anyway, but you should be precautious and change them anyway.
As for LandChad.net, this is spilled milk, although it might be optimal
stategy to compile a brief list of recommended registrars for users as
choices just to avoid potential issues like this where all eggs are in
one basket. The only shortcoming of Epik (aside from being hacked) is
that they were apparently storing password hashed in unsalted MD5, which
is lazy and less secure and simple passwords can be cracked.
Granted, this should also apply to the VPS provider, Vultr. In fact, a
leak of credentials of a VPS provider is potentially much more damaging
because this can mean a data compromise, so it would probably be a good
idea to recommend multiple providers for this as well. Perhaps I'll make
a thread or video on this asking for suggestions.
On 21/09/16 04:13AM, themaster567 wrote:

What should I do for myself and my friends who I've recommended this to? I've done the standard changing my password and adding 2fa, but should I be worried about anything else? From my understanding, the dump includes at least full names, addresses, and passwords. Is it worth it to switch to a different domain service?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/LukeSmithxyz/landchad/issues/132

Their security policies were absolutely unacceptable and any recommendation to use their service should be removed. You should be migrating all services with them to another provider

In my opinion you should apologize Luke. It is not your fault that they were not implementing basic security practices (let alone the kinds that they ought to implement given their focus on hosting controversial content), but you did lead us to them.

That's a bit far. Luke made his recommendation based on what he felt was best at the time. I seriously doubt he could have foreseen this, therefore he did nothing malicious or even wrong. It's just an unfortunate situation.
I'll probably end up transferring my domain elsewhere soon.

Their security policies were absolutely unacceptable and any recommendation to use their service should be removed. You should be migrating all services with them to another provider. If you don't, I think you are a bit foolish.
In my opinion you should apologize Luke. It is not your fault that they were not implementing basic security practices (let alone the kinds that they ought to implement given their focus on hosting controversial content), but you did lead us to them.

You don't have any knowledge of their "security policies." Hashed passwords and plaintext info is the same caliber of leaks that we get continually from much more significant Yahoo! and Google leaks. Epik was targeted not because they had bad security, but for the one reason they are a recommendable registrar: they are a neutral carrier, unlike an increasing number of major US/EU registrars.
(By the way, I'm not sure what it means to "apologize" about something you admit that is (obviously) not only not my fault, but also the best given recommendation given the constraints.)
Regardless, the best strategy in good faith, as I said above, is to have a list of recommendations and as I do some research, I will have a list of registrars. You are invited to volunteer one if you'd like.

As a note for people who aren't concern trolling, it's an abysmally dumb thing to call Epik a registrar of "controversial" content. It's like saying that GoDaddy is a registrar for pornography because they have (doubtlessly millions) of porn sites registered. Epik has like three sites that the media wants to ban and somehow that means they "cater" to "controversial" content just because they don't go out of their way to shut people up. Epik does what all other American registrars used to do: register domains without curation.

Their security policies were absolutely unacceptable and any recommendation to use their service should be removed. You should be migrating all services with them to another provider. If you don't, I think you are a bit foolish.
In my opinion you should apologize Luke. It is not your fault that they were not implementing basic security practices (let alone the kinds that they ought to implement given their focus on hosting controversial content), but you did lead us to them.

You don't have any knowledge of their "security policies." Hashed passwords and plaintext info is the same caliber of leaks that we get continually from much more significant Yahoo! and Google leaks. Epik was targeted not because they had bad security, but for the one reason they are a recommendable registrar: they are a neutral carrier, unlike an increasing number of major US/EU registrars.
(By the way, I'm not sure what it means to "apologize" about something you admit that is (obviously) not only not my fault, but also the best given recommendation given the constraints.)
Regardless, the best strategy in good faith, as I said above, is to have a list of recommendations and as I do some research, I will have a list of registrars. You are invited to volunteer one if you'd like.
As a note for people who aren't concern trolling, it's an abysmally dumb thing to call Epik a registrar of "controversial" content. It's like saying that GoDaddy is a registrar for pornography because they have (doubtlessly millions) of porn sites registered. Epik has like three sites that the media wants to ban and somehow that means they "cater" to "controversial" content just because they don't go out of their way to shut people up. Epik does what all other American registrars used to do: register domains without curation.

The problem is indeed that they stored data in plain-text, and do not appear to have taken the kinds of security precautions that, in my opinion, one should take given the risks they have assumed.
You don't have to apologize, no one can force you to do anything. You can be responsible for leading people to a bad thing, even if you were unaware of it. It seems somewhat unjust but it happens sometimes.

The problem is indeed that they stored data in plain-text, and do not appear to have taken the kinds of security precautions that, in my opinion, one should take given the risks they have assumed.

Unsalted MD5 for passwords is bad practice but still very far from uncommon. What data did they store in plain-text that they shouldn't have?
Besides, with the kind of breach this appears to have been, since other non-database data from the servers has been leaked, using salts probably wouldn't have helped.

The problem is indeed that they stored data in plain-text, and do not appear to have taken the kinds of security precautions that, in my opinion, one should take given the risks they have assumed.
Unsalted MD5 for passwords is bad practice but still very far from uncommon. What data did they store in plain-text that they shouldn't have? Besides, with the kind of breach this appears to have been, since other non-database data from the servers has been leaked, using salts probably wouldn't have helped.

It is not uncommon. That is a problem.
If you might be a target, you should secure your service.
Allegedly, there was a decade old CVE that had already been reported to Epik months before the "breach". The CEO compared the report to "LinkedIn spam". I, personally, will be using a different registrar

Yeah, unsalted MD5 is really bad security practice, and their mail /after/ the big breach didn't give me more trust in them:

At Epik, we take security and the privacy of your information very seriously. Therefore as a precautionary measure, I am writing to inform you of an alleged security incident involving Epik.

Our internal team, working with external experts, have been working diligently to address the situation. We are taking proactive steps to resolve the issue. We will update you on our progress. In the meantime please let us know if you detect any unusual account activity. I am proud of our team’s efforts as we do our part to empower a thriving internet for the benefit of our customers around the world.
You are in our prayers today. We are grateful for your support and prayer. When situations arise where individuals might not have honorable intentions, I pray for them. I believe that what the enemy intends for evil, God invariably transforms into good.
Blessings to you all.
Regards,
Rob Monster
Founder and CEO
Epik Holdings Inc mt

I don't know any registrar with as lax a policy as Epik has, but people using it should really be aware that they can't be trusted on the security side of things, or that you will get the full story from them should anything happen. I'll probably switch to njal.la in the future. That won't be a viable choice for people who "use [their] service in a way that affects anyones health or safety", but they give a better impression when it comes to security and privacy.

Oh boy, Epik's DNS server's are not doing so hot right now. Having partial domain outages along with the inability to edit my host records with getting an internal service error.

I did notice some downtime on Epik's DNS. All my Epik DNSs are resolving fine now, but very disappointing service even if the result of security upgrades. I can understand them disabling DNS edits as a temporary precaution, but it has been several days now, and Epik has not been too transparent about the process.
It might actually be a good idea to have articles on using third-party or self-hosted DNS on LandChad for exactly this reason.
While I will been keeping some domains with Epik, I think I will be moving some of my Epik domains elsewhere. I sympathize with their victimhood here and I am sure they are trying to get through this without normies panicking, but I would certainly appreciate more openness and updates.
Closing this issue since it's not really relevant to this site. I will start a new issue soliciting suggestions for new recommendations.


Support exandable offcanvas navbar

Fix #122
This PR

Implements expanded offcanvas navbar (like this: https://gtrwoz-jghnee.stackblitz.io/) in the UXDS.

Prior to this PR, our only options for navbars were to use (1) an on-canvas toggle navbar that expands to nav links on large screens or (2) a non-responsive offcanvas navbar where navlinks are always hidden offcanvas on all screen sizes.
Option #1 will feel clunky when there are many nav links like GDYT which has 8 nav links one of which is a dropdown.
Testing
Before
https://github.com/CityOfDetroit/COD-Design-System/assets/143553259/d1f41b7c-6c6b-4ad9-97a5-a630c4883897
https://github.com/CityOfDetroit/COD-Design-System/assets/143553259/fd5b7c15-c2d0-45a9-bff7-88032f6deaa3
https://github.com/CityOfDetroit/COD-Design-System/assets/143553259/aa4649e4-f775-4077-9267-3cc2689649e7
After
https://github.com/CityOfDetroit/COD-Design-System/assets/143553259/d7fe57ad-15ad-44aa-8407-4603a8feda3a
https://github.com/CityOfDetroit/COD-Design-System/assets/143553259/b01eade6-eb59-4f88-afdf-1c8c505830df
https://github.com/CityOfDetroit/COD-Design-System/assets/143553259/1ba39717-6788-4a3b-9aed-6afb19ddff09

This is ready to review & merge @jedgar1mx .
Merge conflicts resolved and completed the SASS implementation to support all breakpoints for this feature.