[QUESTION] Merging large JSON fields in response schemas

First check

[x] I used the GitHub search to find a similar issue and didn't find it.
https://github.com/tiangolo/fastapi/issues/701#issuecomment-552312286

[x] I searched the FastAPI documentation, with the integrated search.

[x] I already searched in Google "How to X in FastAPI" and didn't find any information.

Description
I am dealing with JSON data in the order of 10 MBs (up to hundred MBs) which I directly get from a Postgres Instance.
This data is stored as JSONB on this database. To fetch this large amount of data without parsing it into a dictionary I do the following:
items = db_session.query(models.Table.id, cast(models.Table.data, String)).filter_by(id=id).all()

Since I know this data has been properly validated when it was inserted I just use the construct factory from pydantic:
class Item(BaseModel):
    id: int
    data: Union[A, B, C, D]

built_items = [schemas.Item.construct(id=x[0], data=x[1]) for x in items]

Then, on the endpoint I directly return a response using:
starlette.responses.JSONResponse(content=jsonable_encoder(built_items))

But, I still describe the response_model as List[Item] as I need the documentation for this endpoint.
Using this strategy I am able to achieve really good response times, though the original JSON data is encoded now as a string and not as an object when decoded.
So the clients of the API have to decode the response many times:
1) The request itself
2) Once for each JSON object retrieved
Is there any good practice on how to tackle this problem?

When fetching data that's been pre-encoded in JSON, I generally create a custom response class to skip JSON encoding and validation completely. FastAPI will acknowledge response_model in your route for any subclass of JSONResponse, so all you need to do is this:
from starlette.responses import JSONResponse

class RawJSONResponse(JSONResponse):
	def render(self, content: bytes) -> bytes:
		return content

@get("/foo", response_model=MyModel, response_class=RawJSONResponse)
def foo():
    return """{"raw": "json"}"""

The problem I'm seeing here is that you don't get "raw" json from the DB, you get individual JSON elements that still need to be assembled together. What you could always do is use Postgres' json_array_agg() function to aggregate a column of JSON elements into a JSON array. Since id and data are also stored separately, you'll probably also need something like json_build_object('id', id, 'data', data), and then wrap that in json_array_agg.
I don't know how hard that would be to acheive in ORM mode, though.

fastest I achieved is with orjson and transforming my request to json in postgresql, you can do that with something like
class ORJSONResponse(JSONResponse):
    media_type = "application/json"

    def render(self, content: typing.Any) -> bytes:
        return orjson.dumps(content)

    queryjson = """select coalesce(array_to_json(array_agg(row_to_json(response))),'[]')::character varying AS BODYFROM from (
    YOURQUERYHERE
        ) response"""
    response = await db.fetchrow(queryjson)
    return ORJSONResponse(json.loads(response.get("bodyfrom")))

@euri10: I can't imagine why the json.loadstring before re-encoding with osjson would be necessary, unless you're doing that to not skip the validation step, but then I'm not really seeing any benefit to making Postgres handle the JSON encoding instead of just returning your data as rows. Is there anything I'm missing?

I think you're right and that you could directly dump into json
https://magicstack.github.io/asyncpg/current/usage.html?highlight=json#example-automatic-json-conversion
Le jeu. 12 déc. 2019 à 8:37 PM, sm-Fifteen<EMAIL_ADDRESS>a
écrit :

@euri10 https://github.com/euri10: I can't imagine why the
json.loadstring before re-encoding with osjson would be necessary, unless
you're doing that to not skip the validation step, but then I'm not really
seeing any benefit to making Postgres handle the JSON encoding instead of
just returning your data as rows. Is there anything I'm missing?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tiangolo/fastapi/issues/788?email_source=notifications&email_token=AAINSPVV24TJUJM47KQZWDLQYKHGZA5CNFSM4J2ATMG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGXY4JQ#issuecomment-565153318,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAINSPQUKHGCNHS4UM6RS6TQYKHGZANCNFSM4J2ATMGQ
.

Thanks, I'll change my code to what you showed me 👍

I was able to fetch a complete json

When fetching data that's been pre-encoded in JSON, I generally create a custom response class to skip JSON encoding and validation completely. FastAPI will acknowledge response_model in your route for any subclass of JSONResponse, so all you need to do is this:
from starlette.responses import JSONResponse

class RawJSONResponse(JSONResponse):
	def render(self, content: bytes) -> bytes:
		return content

@get("/foo", response_model=MyModel, response_class=RawJSONResponse)
def foo():
    return """{"raw": "json"}"""

The problem I'm seeing here is that you don't get "raw" json from the DB, you get individual JSON elements that still need to be assembled together. What you could always do is use Postgres' json_array_agg() function to aggregate a column of JSON elements into a JSON array. Since id and data are also stored separately, you'll probably also need something like json_build_object('id', id, 'data', data), and then wrap that in json_array_agg.
SELECT json_array_agg(json_build_object('id', id, 'data', data)) FROM mytable WHERE id=5

I don't know how hard that would be to acheive in ORM mode, though.

I used this strategy and it is now really fast 🎊 But now I can't use response_model=List[Item] using Item.construct as the response is already built as a json string.  Is there any other way other than using pydantic construct, but still get the schema documentation on swagger docs?

But now I can't use response_model=List[Item] using Item.construct as the response is already built as a JSON string.

Are you sure? Setting response_model=List[MyModel] works just fine for me.

Apparently I am doing something wrong because fastapi is not skipping validation with your solution. When I run your code exactly like this:
from starlette.responses import JSONResponse
import pydantic

class CustomJson(JSONResponse):
    def render(self, content: bytes) -> bytes:
        return content

class Item(pydantic.BaseModel):
    id: int

@app.get("/foo", response_model=Item, response_class=CustomJson)
def foo():
    return """{"raw": "json"}"""

I get this error:
File "...\lib\site-packages\fastapi\routing.py", line 72, in serialize_response
    raise ValidationError(errors, field.type_)
pydantic.error_wrappers.ValidationError: 1 validation error for Item
response
  value is not a valid dict (type=type_error.dict)

Apparently, response_class is not actually doing anything so I have to manually create the CustomJson instance. Is this a bug?
from starlette.responses import JSONResponse
import pydantic
class CustomJson(JSONResponse):
def render(self, content: bytes) -> bytes:
return content
class Item(pydantic.BaseModel):
id: int
@app.get("/foo", response_model=Item, response_class=CustomJson)
def foo():
return CustomJson("""{"raw": "json"}""")

@littlebrat This isn't a bug -- FastAPI first takes your return value, then converts it into the specified response model, then feeds that to the response_class.
The idea is that the custom response class should expect to receive a model, rather than raw bytes.
You can see the logic for this here:https://github.com/tiangolo/fastapi/blob/7a445402d4960d6173d76dac43393ad6c5040521/fastapi/routing.py#L126-L146
Currently, you can get around this by manually returning the response class, which short circuits any processing. (You could put this into a small decorator if you want easier reusability.)
If you think there is a gap about this in the documentation, a PR would be welcome!
Also, if you think the behavior should be different, feel free to open a feature request issue.

Apparently, response_class is not actually doing anything so I have to manually create the CustomJson instance. Is this a bug?
from starlette.responses import JSONResponse
import pydantic

class CustomJson(JSONResponse):
    def render(self, content: bytes) -> bytes:
        return content

class Item(pydantic.BaseModel):
    id: int

@app.get("/foo", response_model=Item)
def foo():
    return CustomJson("""{"raw": "json"}""")

No, you're right, I forgot about that part, my bad. You actually need to return a pre-instanciated Response object.

Thanks everyone for the help here! :rocket: :cake:
Does that solve it for you @littlebrat ?

Sorry for the delay, yes it did!

Thanks for reporting back @littlebrat ! :rocket:


Adds DLX

🚧
This MR

adds DeadLetterExchange (DLX) support ()
performs some code prettyfying for dispatcher and workerTemplate (dddf49734c2b97c838a97c17d279f2692b582c1d)

dead letter seems to work now but we still have to fix broken job message when an error occurs.
see todo ...

Error handling is now implemented as follows:
Workers will return a promise in their main method which, in case of an error, throws.
This will be catched in the workerTemplate. All messages from workers that could be read will be acknowledged, while only messages in the jobs and results queue will forward the message to the dead letter queue in case of an error by calling nack with specific arguments.
So if a job fails in e.g. the third of total five steps, all queues will be empty afterwards except the "DeadLetterQueue".
This queue will contain the failed job with the current achieved status and the error that has arised.
So Jobs that have failed will not get retried and will not remain in the job queue. Instead, they are moved to the deadletter-queue.
@JakobMiksch please review


Webpack build

[ ] Before merging, check the install via symlink in another web app to verify the prepare script is working


grab bag of feedback

This is neat! Some quick thoughts after a few minutes of poking around. I really like it, so please don't take any of this feedback as negative. It's much easier to list the stuff that stood out rather than comment on the overall package, which is excellent. Thanks for putting this together!

That output is awesome. Any reason to include the random # in the line prefix? I'd think the pid would be enough and it would be great to reduce noise.

Is this using a shell, or running the process directly? This makes a big difference when writing commands and can often result in unexpected behavior, especially when trying to use redirects or pipes. Which is quite common in my experience. :)

Actually, that touches on a larger issue. I think it should be possible to just do:
cmd.execute("sudo apt-get install xyz > /tmp/output.txt")
instead of:
cmd.execute(:sudo, "apt-get", "install", ...)
This is the most common use case, in my opinion, and supporting this pattern will encourage people to stop using bash to write this stuff.

Escaping can be a challenge with the current syntax. What should this do?
cmd.execute(:ls, "hello world")

[71555-047a57b2] Running ls hello world
[71555-047a57b2] 	ls: hello: No such file or directory
[71555-047a57b2] 	ls: world: No such file or directory
[71555-047a57b2] Finished in 0.002 seconds with exit status 1 (failed)

Any reason to use symbols vs. strings for the command name? Sadly some commands have spaces in them...

I think that commands should throw exceptions if they fail by default. This is one of the biggest design flaws in bash, and I can't tell you the number of problems it's caused me (and people who work with me) over the past decade. When I write scripts I almost always want to halt on failure. I recognize that if we make this change we can't use execute!, sadly.

Timeout looks really handy. I've definitely wanted this a few times.

I think you could probably ditch the with exit status 0 (successful) and only log failures. That will make them stand out more, I think. Or at least make successful use the same color as the rest of the line.

Features I think you can safely ditch:
-- env: (this can easily be handled with ENV)
-- chdir (easily handled with Dir.chdir("x") { ... }
-- redirection (typically in the command, not needed in ruby land)
-- :user/:group/:umask (that's what sudo is for)

Can I convince you to change execute to run? I love brevity and there will never be a chance to change it after this! :)

Here's a typical example, one of hundreds that I have from my old startup. Think about this snippet - would it make sense to separate out every argument? Would it make sense for the script to continue after failure? Would it make sense to replace shell redirects/pipes with a custom syntax? Just some food for thought.
  # restore db
  def db
    if Util.mac?
      chmod, dst_dir = "#{user}:admin", "/usr/local/var/mongodb"
    else
      chmod, dst_dir = "mongodb", "/var/lib/mongodb"
    end

    Util.rm_and_mkdir(TMP)
    Dir.chdir(TMP) do
      # untar
      banner "Untarring #{db_src}..."
      run("tar xvfz #{db_src} --exclude 'prealloc*'")

      # atomically move the files into place
      with_db_stopped do
        Dir.chdir("mongodb") do
          Dir["startup*"].sort.each do |src|
            next if src == "lost+found"
            next if src =~ /\.old$/

            dst = "#{dst_dir}/#{src}"
            old = "#{dst}.old"
            run "sudo rm -rf #{old}"
            run "sudo mv #{dst} #{old}" if File.exist?(dst)
            run "sudo chown #{chmod} #{src}"
            run "sudo mv #{src} #{dst}"
            run "sudo restore > /tmp/restore.txt"
          end
        end
      end
    end
  end

Happy to help implement any/all changes... Thanks again!

Thanks, that's great feedback!
Few initial thoughts:

I agree the uuid should probably be shorter. I have followed the same principle that is used for creating tempfile names in ruby, though I'm not sure that PID is the one I will choose as you can run many commands and they all may run in the same process but it would be good to have unique identifiers per command executed. Also, I want to provide ability to specify different printers apart from the current one such that skip printing the uuid all together and still colour the output.
Basically the execute relies on spawn similarly to exec, system and open3 Ruby commands. This means execution in new process environment that isolates the command from side effects. Which also gets all the environment variables and options passed down.
Sure you can run the command the way you have mentioned. The readme is definitely not comprehensive:) If you run cmd.execute("ls -la > output") it will respect the redirect and execute directly the string. There are many ways to pass the arguments which try to mimic the spawn behaviour.
You can use both symbol & string, makes no difference. The reason why I provided the examples this way is simply a stage setup for more ruby like syntax. I was intending to add ability to basically run cmd.ls "-la" and the command just do what is intended. Agree it won't work in a lot of cases given complex arguments or executables but it is a nice Ruby-like way to run simple stuff if someone needs to.
I think designing api we need to be careful assuming too much. I completely understand the deficiency of bash but also believe in providing the means for handling the rough edges to the user. Raising error is not always the best way to handle application logic flow. I can easily see circumstance where a person will want to examine exit status and then decide whether raising error is what's needed or maybe simply aborting script with some useful debug info. I find raising errors for a generic library too rigid. Hence, I would still think of having dangerous version of the command running. Best practice can be explained in readme and ways to handle it?
I'm not sure about ditching the features. I think constructing a uniform commands out of options and properly escaping these is quite important. For example :group option is tricky to escape as part of larger command and if the tested code can do it for you so much better. The :env does quite few things for you including transformation of symbols into proper environment variables. Again, I think if the library can provide value in constructing these I think it should, especially when user can focus on writing Ruby?
I'm not dead set on execute vs run, we can defo come back to this point once we clear up all the other issues.

On another note, it would be great to open up separate issues for each bullet point. Then we are able to reference commits against features. I should be able to find some time tomorrow.

I've moved all the feedback to separate issues.


Test and changes for CASSANDRA-11255

A new test for CASSANDRA-11255 and some changes due to a new default precision for double values in COPY TO.

+1


Raycast.wurst

Contains
For 2D:

[x] point projection onto ray
[x] ray X line
[x] ray X ray
[x] ray X segment
[x] ray X circle

For 3D:

[x] point projection onto ray
[x] ray X plane
[x] ray X disc
[x] ray X sphere
[x] ray X triangle

Of course, there are lots of extra things that can be implemented but they can be surplus in Wc3 for stdlib.

The 3d functions look a bit messy, but other than that good stuff, though on the brim for stdlib.
You should recheck doc for typos "if it behind" and add some package doc where you explain the result tuple and perhaps a small wc3 use case.
Sry for taking some time to check up.

add some package doc where you explain that functions return the result tuple

Can I add the hotdoc before package Raycast at the top of the file?

though on the brim for stdlib.

There are several functions that I haven't dared to add
For 2D:

ray X rectangle
ray X convex polygon

For 3D:

ray X box
ray X cylinder (cone)

Do these fit stdlib? Are these overkill?

Can I add the hotdoc above package Raycast at the top of the file?

I prefer it after the package and import directives. You can use normal /* */

Do these fit stdlib? Are these overkill?

Probably overkill, also depends on size of course.

There's some map example if it could be useful. Dunno why.

ain't sure how it can be incorrect. Did you mean that

Yea that's good. Last thing I'm thinking about is whether "success" is the correct wording, since nothing really succeeds here. if ray.intersects sounds better than if ray.success to me, so that might be better?

I think you're right. success would be better for functions like .intesectSomething but not for raycasting where any result is basically succeed.

I think except for spaces-tabs issue this looks good now 👍

Whitspaces replaced with tabs.


Additional relationships

Hello,
I was working with the ATT&CK data sources repo along with this one and noticed that some of the relationships I wanted to use weren't defined here. I've created them and wanted to contribute back! I also made some small edits to a couple of existing relationships where there appeared to be a typo.
Added

process accessed file
process accessed windows registry key
process created thread
process loaded driver
user accessed file
user accessed AD Object
user accessed Windows registry key
user requested AD credential

Modified

user attempted to authenticate from ip
user attempted to authenticate from port
user attempted to authenticate to computer

These relationships had event 4625 listed twice; I changed one of those duplicate entries to 4624. I also renamed "user attempted to authenticate to host" to "user attempted to authenticate to computer" based on the language in the ATT&CK data sources repo. If that's not a welcome change or I've misunderstood please let me know!
I hope these contributions are helpful!

Hello @reidgilman!!
Sorry for the late reply, thank you for submitting your first PR to the OSSEM-DD repository 🙏  We will review it and continue the conversation about it 😃

Hey @reidgilman !!
Here some comments for your PR:

relationships/process_accessed_file.yml: Okay.

process accessed windows registry key: Okay. I removed events 12,13, and 14 because we are considering those events in the created, deleted and modified relationships, which represent the main goals of these events.

process created thread: Okay.

process loaded driver: We consider Sysmon 6 as a potential event for the driver metadata component because the event only gives us context of the driver being loaded. There is no Process context within the event data fields. I am removing this file, but we will definitely continue the conversation about it and identify another event or table that represents this activity 😃

user accessed file:  Okay

user accessed AD Object: Okay

user accessed Windows registry key: Okay. I removed events 12,13, and 14 because we are considering those events in the created, deleted and modified relationships, which represent the main goals of these events.

user requested AD credential: Okay

user attempted to authenticate from ip: Okay

user attempted to authenticate from port: Okay

user attempted to authenticate to computer: We will get back to you about the "computer" data element. We need to review this concept and define which term would align with OSSEM Common Data Model.

Are you in our discord server? If not, please use the following link (https://bit.ly/OTRDiscord) so we can continue the conversation about your PR. Thank you again for your contribution to the project 💜 We look forward to continue collaborating with you and improving and adding more relationships to the project 🍺
Best,
Jose

Thanks @Cyb3rPandaH! I don't feel particularly strongly about the host/computer distinction. My use case started with ATT&CK so I was thinking in and using their terminology.
I was using OSSEM-DM and OSSEM-DD to see which logs might be applicable to a set of ATT&CK techniques. I was surprised there was no relationship matching user authenticated to computer (https://github.com/mitre-attack/attack-datasources/blob/5812c1f8051ad2bc8d7a76a983dbb7fb5044f0ca/contribution/user_account.yml#L79).
Does host map on to an element of the Common Data Model? I have really only used OSSEM-DM! :)


OCPBUGS-24444: Check topolvm only when it should be deployed

/assign @ggiguash

/jira refresh

/lgtm

/hold

/unhold

/retest
ci/prow/microshift-metal-tests-arm
Error: Timed out waiting for pod image(s) from the 'openshift-storage' namespace to be downloaded

/cherrypick release-4.15


[#170048540] refactor docker-compose

Warnings

:warning:
Please include a description of your PR changes.

Affected stories

⚙ #170048540: Rifattorizzare docker-compose

  Generated by :no_entry_sign: dangerJS against 59e248aa95cb8f171fd0afa7c0703399395b32ed

@gunzip, please also update the env variables in the README file accordingly.


evictRandomBytewise with worst-of-2 eviction

This PR adds the ability to limit the mempool size to a number of bytes, using the -maxmempoolbytes command line option. It adds a method to pick two random transactions from the mempool, weighted by the transactions' sizes, and evict the transaction with the lowest fee/kB. Transactions that are prioritized will be skipped.
The -maxmempoolbytes option defaults to 50 blocks' worth of transactions. It uses the transaction's serialized size, not the in-memory size. The in-memory size is typically 2.5x higher. Thus, with 1 MB blocks, the default setting allows for approximately 2.5 * 50 * 1 MB = 125 MB of memory usage by the mempool.
Performance-wise, each eviction typically takes less than 200 us on a Core i7 4790k. Performance should be roughly O(1) vs mempool size.

Code looks good! unACK.

+1 to removing the timing code-- if I recall correctly, some versions of boost on some platforms have issues with nanosecond-resolution clocks.
I'd also vote to remove the -debug=mempool detail on which candidates are compared; I'm sure it was useful during development, but I doubt it will ever be useful again now the code is written and debugged.
Other than that untested ACK.

I'm pretty concerned about this approach. It is weak on policy application, because it mainly favors the eviction of large-size transactions, and not transactions that are large relative to the fees they pay.
The single-pair fee comparison is an extremely weak and non-deterministic way of awarding those who offer higher fees a greater chance of relay and confirmation.  We must do better because this is the primary job of the bitcoin network, and the most fundamental service that is offered in exchange for bitcoin.
Remember, randomized eviction is just an implementation cheat - not a policy.  With XT's emphasis on the importance of zero-conf, introducing mempool inconsistency across the network is not good at all.
I have an alternative which uses boost multi-index to apply a deterministic policy that is defined in the index comparator function, without the weird complexities in the core solution.  I'll have it submitted within 24 hours for your consideration.

@dgenr8 can you please post your objection to the idea (i.e. its not a code-review) to the mailinglist? Discussing concepts on a merge requests doesn't make sense to me.
Also, we have been discussing the approach for weeks now, with various patches being send to the ML in the mean time. Why didn't you voice your ideas then?

@zander I'm sorry, as usual I thought I would be done much sooner.

Yep, let's take the design discussions to the list and keep this thread for code review.
@dgenr8 Could you re-post your concerns to the list to start a thread? Copy/paste is fine.

@dgenr8: I posted a reply to your comments on the bitcoin-xt list: https://groups.google.com/d/msg/bitcoin-xt/9_YHacVC-b4/VtRI0obBEAAJ

Okay, @gavinandresen candidate comparison debugging lines removed. @zander Timing code removed. @mikehearn Command-line argument is for in-memory bytes (using 2.75x vs serialized bytes as a magic number).

I am testing builds of this branch (before the removal of debugging) on two of my nodes (nix and mac os), It's been running smoothly for 24 hours with a reasonable production default  of maxmempoolbytes=400411697   Tomorrow I will be switching one of them to non reasonable defaults in  a production environment and will watch the mempool tail closely

Thanks for the testing, @rromanchuk

Hey guys, i've been struggling to get the branches of code i need to run in order to successfully test this. I just took a quick peak at the diff and realized maxmempooltx is tightly coupled with maxmempoolbytes so it might be helpful to clarify in the command help copy. Looks like using an unreasonable maxmempoolbytes setting, of let's say 10mb, the eviction code may never run if maxmempooltx is set to something like 50000, or may not run as anticipated by the user.
Just started putting this to work by temporarily squeezing maxmempooltx downwards.

@zander and @dgenr8 noted that the empirical factor borked the default value. It's unborked in 8d51bcc. 8d51bcc tested with default maxmempoolbytes value after a few hours:
$ for i in `seq 1 5`; do echo `./bitcoin-cli getmempoolinfo`; sleep 4; done
{ "size" : 5665, "bytes" : 49988347 }
{ "size" : 5676, "bytes" : 49994836 }
{ "size" : 5684, "bytes" : 49997734 }
{ "size" : 5690, "bytes" : 49985555 }
{ "size" : 5697, "bytes" : 49987535 }

Update: I have been running this on two different production nodes without any problems and working as expected. I also tested this with extreme/nonsensical settings on both test and mainnet without any issues to report.

The size-weighted limiting has a huge problem: it doesn't stop relaying when the mempool is full.
This is a direct result of the fact that it doesn't implement a way of prioritizing transactions.  It attempts to treat them all equally in relaying to miners, even though miners are acknowledged to prioritize by feerate.

The size-weighted limiting has a huge problem: it doesn't stop relaying when the mempool is full.

Well, not if you're using maxmempooltxns
I'll take this to the discussion group, but as a user, I much rather relay spam and evict good actors during a "spam attack" than having my node crash. Node operators shouldn't have to constantly tweak minrelayfees just to keep their nodes running.

@dgenr8 : I think that regulating the size of a node's mempool and regulation of the network's bandwidth are very separate things, and should operate under different policies. A node can run out of memory long before bandwidth, or vice versa. Miners, for example, currently want to run with a mempool of about 4 MB due to limitations in getblocktemplate, but they shouldn't stop relaying transactions when they get to that level.

I've mined at least one block with this code on mainnet.

@jtoomim If we go this way, do you currently know of a reason not to merge this into a release for general testing?

I know of no obstacles. As I've said, I've used this on mainnet for months.

@dgenr8 i've been using this since day 1. Very well tested against edge cases and tested against spam attacks during the testnet bip101 tests.  Probably not as much on everyone's mind since the last spam attack.

Closing in favor of #90.


Trailing comments get highlighted as active syntax

It would be nice if the grammar could recognize trailing comments the way it does non-trailing ones.
# sole comment   ← monotone color

ct state invalid drop    # trailing 123 comment

                          ↑ looks like syntax ↑

P.s. Thanks for creating this language addon, much appreciated. :+1:

Thanks!


Update ladd ledit diagrams

Codecov Report

Merging #385 (be5ab07) into master (c2023be) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #385   +/-   ##
=========================================
  Coverage     68.40%   68.40%
  Complexity     1131     1131
=========================================
  Files           135      135
  Lines          3865     3865
  Branches        521      521
=========================================
  Hits           2644     2644
  Misses          996      996
  Partials        225      225

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c2023be...be5ab07. Read the comment docs.


Add focus state to menu button

Trello card
Trello 5109
Context
Silktide has flagged an accessibility issue with the new menu button on mobile. It doesn't change state on focus.
Changes proposed in this pull request
Add a focus state to the menu button.
Guidance to review
Tab through the homepage with the website rendered as if on mobile or tablet and check that, when the menu button is in a focused state, it has a black background, white text and icon and a pink underline.

Looks good to me @garethwilliamsdesign - happy for this to be merged once it has had tech eyes on it from @StackedPancakez


chore: improve pre commit hook and prettier checks

Description
Implement pre-commit hook the intended way
Why
The current pre-commit hook is poorly implemented because

prettier and linter are always running on all files not only staged ones
files modified by prettier are not added to the initial commit thus requiring an additional commit
unit tests are not executed on staged files

this commit solves all the issues implementing the pre-commit hook the intended way nevertheless we also added pretty checks to the PR check pipeline breaking it in case of unformatted files.
Issue
n/a
Checklist
Please delete options that are not relevant.

[x] I have followed the contributing guidelines
[x] I have performed a self-review of my own code
[x] I have successfully tested my changes locally
[x] I have added tests that prove my changes work
[x] I have checked that new and existing tests pass locally with my changes

why are there so many new packages/dependencies added?

Only one dev dependency has been added - lint-staged. But it comes with many direct and indirect dependencies. This is a package that is only needed on the developer machine because it is executed before the commit. Maybe it's better to install this globally on the machine because we don't need to download all those packages in the build pipelines.

why are there so many new packages/dependencies added?

transitive from lint-staged package

Husky now executes lint-staged through npx which avoids installing it and dependent packages during the pipeline runs.


Terminology function: property

collection<Coding|CodeableConcept> -> property(code: string) : collection<String|Integer|DateTime|Decimal|Coding>

This function takes a collection of Coding or CodeableConcept elements as input, and returns a collection containing the values of the specified property.
Example:
Patient.reverseResolve(Condition.subject).code.property('effectiveTime')

We could potentially pre-adopt this R5 expand feature to enable a more efficient implementation: https://ontoserver.csiro.au/docs/6/ext-r5-preadopt-exp.html

Following the implementation of #757, I think lookup will be the best way to implement this.

To simplify the implementation we are going to add the optional second parameter that specifies the type of the property being requested, one of: code | Coding | string | integer | boolean | dateTime | decimal
collection<Coding|CodeableConcept> -> property(code: string, [type:string]='string') : collection<String|Integer|DateTime|Decimal|Coding>

@johngrimes : does it make sense to have this function on CodeableConcept. I mean we can definitely combine the properties for all codings but is that useful in any way?

Added in v6.1.0.


Update tutorial.md

added + removed words and changed sen. structure for clarity

Learn Build status updates of commit ab3a370:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

data-explorer/kusto/query/tutorial.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

@shsagir

Can you review this PR?
IMPORTANT: When this content is ready to merge, you must add #sign-off in a comment or the approval may get overlooked.

#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

#please-close


Roll Engine from d5cadd28b0bf to 48df97ab4e9d (2 revisions)

https://github.com/flutter/engine/compare/d5cadd28b0bf...48df97ab4e9d
2021-11-11<EMAIL_ADDRESS>Roll Skia from a583a0fdcc15 to 6fae0523629f (2 revisions) (flutter/engine#29661)
2021-11-11<EMAIL_ADDRESS>Roll Dart SDK from 38e7078fa2b7 to e1a475a40934 (19 revisions) (flutter/engine#29659)
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Please CC<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose
To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

Commit queue failed; closing this roll.


pacbio/tubes migration from VueX to Pinia

Defined stores folder for Pinia
Pinia store defined for pacbio/tubes - src/stores/pacbioTubes.js added

Actions, getters method refactored for Pinia
Mutations removed
Unit test cases added

VueX store for pacbio/tubes removed - src/store/traction/pacbio/tubes

Unit test cases removed

@BenTopping @stevieing I couldn't find any mutations/getters for this Vuex store being called from anywhere else. Please correct if otherwise and I will make the required changes.

@BenTopping @stevieing Thank you for the comments. I will work on this.


Contributors Link fixed and Games Link added

PR Description 📜
The contributors link in the Navbar was not working, and the games link was missing. I reviewed index.html to identify the issue, corrected the contributors link, and added a new link called "Games" in the navbar.
Fixes #4818

Mark the task you have completed ✅

[ ] I follow CONTRIBUTING GUIDELINE & CODE OF CONDUCT of this project.
[ ] I have performed a self-review of my own code or work.
[ ] I have commented my code, particularly in hard-to-understand areas.
[ ] My changes generates no new warnings.
[ ] I have followed proper naming convention showed in CONTRIBUTING GUIDELINE
[ ] I have added screenshot for website preview in assets/images
[ ] I have specified the respective issue number for which I have requested the new game.

Add your screenshots(Optional) 📸
https://github.com/user-attachments/assets/48e265d7-df12-4c87-8bbe-2c06fe57795b

Thank you soo much for contributing to our repository 💗

Thank you @shreyasmunge ,for creating the PR and contributing to our GameZone 💗
Review team will review the PR and will reach out to you soon! 😇
Make sure that you have marked all the tasks that you are done with ✅.
Thank you for your patience! 😀

hey,
This issue of contributors link not working along with others is already addressed in the PR #4802

@kunjgit and @ishita-43 I have resolved the merge conflicts, Please have a look .

hey, did you check how navbar responds for adding new element  ,for different screen sizes . you can check that by right click then  going to inspect and pressing ctrl+shift+M , and then draging the screen using || on RHS of screen to see response of the navbar.
According to me one extra element will cause overflow of elements in navbar for small screen size

Thank you  @shreyasmunge , for your valuable time and contribution in our GameZone 💗.
It’s our GameZone, so Let’s build this GameZone altogether !!🤝
Hoping to see you soon with another PR again 😇
Wishing you all the best for your journey into Open Source🚀

@ishita-43 Please add  Level  Label to this PR.


ImportError: Keras requires TensorFlow 2.2 or higher. Install TensorFlow via pip install tensorflow

ImportError: Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 64, in
from tensorflow.python._pywrap_tensorflow_internal import *
ImportError: DLL load failed: The specified module could not be found.
Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/errors
for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.
During handling of the above exception, another exception occurred:
ImportError                               Traceback (most recent call last)
 in
16
17
---> 18 from keras.models import Sequential
19 from keras.layers import Dense, Dropout, BatchNormalization, Activation
20 from keras import regularizers
C:\ProgramData\Anaconda3\lib\site-packages\keras_init_.py in
4 except ImportError:
5     raise ImportError(
----> 6         'Keras requires TensorFlow 2.2 or higher. '
7         'Install TensorFlow via pip install tensorflow')
8
ImportError: Keras requires TensorFlow 2.2 or higher. Install TensorFlow via pip install tensorflow

Please import keras from tensorflow 'from tensorflow import keras.
See https://github.com/keras-team/keras/releases/tag/2.4.0

@pratikshambharkar
Can you please share the complete stand alone indented code for us to replicate the issue faced or if possible share a colab gist with the error reported.
please refer to this link as per the error reported.
You may also verify with existing issues containing "ImportError: DLL load failed": #42495 #43026 #43003

Correct, Conda is not officially supported and often lags behind the current release.
As @bhack noted, there's a mention here: https://www.tensorflow.org/install/pip#conda —but little appetite for further visibility of an out-of-date package. But you're welcome to add a an entry to the errors.md page, if you think that is useful. Thanks

@Saduf2019
i have tried this also :- https://docs.anaconda.com/anaconda/user-guide/tasks/tensorflow/
< conda create -n tf tensorflow
conda activate tf
conda create -n tf-gpu tensorflow-gpu
conda activate tf-gpu

import tensorflow


fix: missing nccl

The first CUDAExtension is missing the libnccl.so, we should add nccl to the libraries argument. Otherwise, when import bmtrain, it will complains
/site-packages/bmtrain/nccl/_C.cpython-38-x86_64-linux-gnu.so: undefined symbol: ncclBroadcast

As the described problem in #103 and OpenBMB/CPM-Bee#18

Fix since PR #138.


Fixes NPM install via github

The current main path is unresolvable when installed via NPM.

thanks for the PR @blader ! We're working to make the JS SDK ready for npm


slide-toogle : change not fired with drag'n drop

Bug, feature request, or proposal:
Bug
What is the expected behavior?
with
<md-slide-toggle [(ngModel)]="test" (change)="printTest($event.checked)">
I expect
printTest() is called with click or drag'n drop
What is the current behavior?
printTest() is called on a click BUT NOT on drag'n drop. In both cases, the value "test" is updated though.
workaround :
 <md-slide-toggle [(ngModel)]="test" (ngModelChange)="printTest($event)">
Which versions of Angular, Material, OS, browsers are affected?
2.0.0-alpha.9-3 , last chrome or firefox

This should be fixed with https://github.com/angular/material2/commit/0b5b6f2fda62a4977f03c761e02ff2eb934cf52f, but is not released yet.


Untranslated text in pop up notification

Notification text has not been extracted for translation: https://i.imgur.com/6rznWq4.png
Version 1.0.6, server version 1.1.2 on Android downloaded from the app store.
Based on FR- QA

Tab :Additional Translations Mar 15- Ready has the translation
Related to issue 582

Translation done and added to tab: Additional Translations Mar 15- Ready in each language file


Liks to "Edit on GitHub" are broken for some doc pages

If you click on the "Edit on GitHub" link, it takes you to a 404 GitHub page for some doc pages:

thanks for reporting this! we are moving to jupyterbook, so I'm closing this


files option does not respect working-directory setting

files option does not seem to respect jobs.<job_name>.defaults.run.working-directory from github. I have to prepend working directory path to every file under files for gh actions to find the correct file.

here is an excerpt, hope this helps:

jobs:
  build_and_deploy:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: ./frontend

    steps:
      - uses: actions/checkout@v4

      - uses: pnpm/action-setup@v4
        with:
          version: latest
          run_install: false

      - shell: bash
        run: |
          echo "STORE_PATH=$(pnpm store path --silent)" >> $GITHUB_ENV

      - uses: actions/cache@v4
        with:
          path: ${{ env.STORE_PATH }}
          key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
          restore-keys: |
            ${{ runner.os }}-pnpm-store-

      - run: pnpm i

      - name: Build
        run: pnpm run dist

      - name: Deploy to GitHub release
        uses: softprops/action-gh-release@v2
        with:
          files: frontend/dist/*.zip
          tag_name: zipped_dist

where pnpm run dist command packages everything into a zip file.
If this workflow looks good to you, I can try to provide a minimal reproducible  repo.

I think I see the issue
the following settings the working directory for each step. think of it like the PWD path when executing a step
    defaults:
      run:
        working-directory: ./frontend

In your files declaration you provide the patternfrontend/dist/*.zip which will be resolved while you are in the frontend directory which would assume a path more like frontend/frontend/dist/*.zip because its relative to the working-directory for the step.
      - name: Deploy to GitHub release
        uses: softprops/action-gh-release@v2
        with:
          files: frontend/dist/*.zip

Could you try and remove the frontend prefix from that step? Since the working-directory is frontend that would then resolve to something like  frontend/dist/*.zip
      - name: Deploy to GitHub release
        uses: softprops/action-gh-release@v2
        with:
          files: dist/*.zip

afaik,
        with:
          files: frontend/dist/*.zip

is working (current working version of my workflow), but
        with:
          files: dist/*.zip

does not work (cannot find anything matching *.zip) (my initial not-working version)

What happens when you remove the frontend part of the path after setting the working directory to frontend?

I am having the very same issue. In my case is pretty much the same but is a rust application.


txn: check key commited from store if lock not found

Issue #328
Do not let tidb report partial commit error log during benchmark tests

@coocood  PTAL

LGTM


Syntax Issue

File "filemovergui.py", line 45
".wmv",".mov",".qt",".rm",]"CodePerfectPlus",
^
SyntaxError: invalid syntax
When I run python3 filemovergui.py

Thanks, @Raavtube
For pointing out Bug. It's now fixed.


code coverage -> codecov.io

Setup code coverage similar to CoreNEURON <- @ohm314

Update Jenkins CIs with GitHub username alexsavulescu
Retest this please

Update Jenkins CIs with GitHub username alexsavulescu
Retest this please

Coverage report here:
https://codecov.io/gh/BlueBrain/nmodl/tree/28093a6b15b4ea4177b1082c3fc1cd59f0d5b8f9

Coverage report here:
https://codecov.io/gh/BlueBrain/nmodl/tree/28093a6b15b4ea4177b1082c3fc1cd59f0d5b8f9

Badge is remaining though!

Badge is remaining though!


⚠ Samsung has degraded performance

In d724443, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2802 ms

Resolved: Samsung performance has improved in 762235c.


Module hyprland/language cannot be configured to be usable

I started trying to configure the hyprland/language module and came up absolutely flat. The only string I can get out of it is the full long description from the keyboard layout, and that is useless

For two of the layouts I use it is so long as to take up half the bar
It includes spaces, dashes, and other characters that I can't seem to match in the config so I can't even add a per-layout override in the config file
There is no short option as in the sway equivalent
There is no flag option as in the sway equivalent

The final output I want is a flag, but the flag option probably won't help me even if implemented correctly because, for example, I want the Kazakh flag for my Russian layout. What I really need I think is just the short identifier used to set the layout in the first place. That's a simple code I can use in the config to add custom label string per-layout.
I looked into fixing it myself, but the C++ involved is going to be out of my league especially considering the comparable Sway module is also reported not to work properly so I can't just copy bits (#1501, #1909, #1536)

Just a little more info looking into this, lets take one of my layouts as an example: My dvp layout (that's the short key used to set it, whether with setxkbmap or hyprctl) reports its full name as "USA - Programmer Dvorak`.
Since there are no options other than {} for the format string that's the only string I have to work with, but the format-* key that could be used to customize this is using the "brief" name behind the scenes. The brief layout is always being reported as an empty string by the hyprland/language module.
I can actually change the string with this:
        "format-": "DVP",

But that of course changes it for all of my layouts because they all parse to an empty string for brief.

nc -U /home/user/tmp/.hypr2.sock | while IFS='>,' read event noop dev layout; do
  if [ "$event" = "activelayout" ]; then
    case "$layout" in
      Czech*) echo "🇨🇿";;
      English*) echo "🇺🇸";;
    esac
  fi
done

it requires the hyprland ipc socket2 to be available at ~/tmp/.hypr2.sock (I link it there from the default location by other script). It could probably leverage IPC from waybar somehow but I haven't investigated that far, this works for me (tm)

Thanks for the hack. I might cobble something together for my use case too because this still isn't fixed and the status-quo is not usable in the slightest.

This works for me:
    "hyprland/language": {
        "format": "{}",
        "format-en": "🇺🇸",
        "format-de": "🇩🇪",
    },

So my ugly script stopped working some time ago so I gave the module a chance again and managed to configure it to behave as I want it to using this settings:
    "hyprland/language": {
      "format": "{}",
      "format-cs": "🇨🇿",
      "format-en": "🇺🇸",
      "keyboard-name": "at-translated-set-2-keyboard"
    },

Not sure if the keyboar-name key is needed and I had hard time figuring out the "short suffix" after format-. It's confusing, in hyprland config I've got us,cz layout but here for some reason I must use en for the US layout. Found out basically by trial and error :/
Maybe this is now more of a documentation issue? It would help to update the wiki, adding a way to find out what the "short" name for the layout is - I couldn't find it anywhere.

@jficz I've had the same issue with trial and error. I noticed after your comment that using waybar --log-level debug will show the correct matching keyboard name to use, but not the right short code to use in the language. I added some more debug outputs (and hence identified that my bespoke keyboard layouts don't even have short codes) that show a lot more about what is going on.


User getting access to other users session variables

We're using DistributedSqlServerCache and in our testing we keep on hitting a very serious problem where two users coming in through IIS, in the same time period, have their sessions mixed up giving them access to each others session variables. We're using .net core 1.0.1.
I've read through the logs to try to find any clues. Googled my way through tons of materials, looking at our configuration but not found anybody with exactly this problem.
Any hints?

/cc @Tratcher

What I'm currently looking into is that this behavior would maybe happen if the SqlServerCache was not really enabled but using the memory cache that would then be shared depending on which worker process the user was coming in through.  I found that we're doing services.AddMvc(); before services.AddDistributedSqlServerCache(); I'm not sure if that changes anything but next test tomorrow will be doing services.AddMvc(); after adding the cache.

AFAIK the middlewares order is very important

BTW your middleware order should be like this
app.UseSession();
app.UseMvc();

The session IDs are GUIDs sent as encrypted cookies to the client. Nothing except bad response caching should cause one user's session cookie to be given to another user. I assume you've verified that these users are coming from different machines, as opposed to one user just signing out and the other signing in via the same browser?

Thanks @Tratcher.  Yes, the users are coming from two different companies. This is a highly unusual scenario we discovered because of another unusual bug. The frontend is a single page Polymer/Web Components app.

User A logs on and start to continuously do activity in order to trigger this bug.
User B send a request to the API, initiates a call to complete the login.
User B writes information to the session, a company id
User B hits an Crypto/OCSP bug that causes a 15-20 second holdup
User B reads the company id from the session and it gets the wrong company id of User A instead out of the session!
User B request completes with an error as a check on our side reveals the inconsistency.

So the session does seem to mixup underneath the users in the middle of a single request.

How readily can you reproduce this? Can you create an independent repro app?
I'd start with a Fiddler trace from both clients to verify the cookies. Then I'd want to see the server logs for those requests.
Steps 3 and 5 happen on the same request? Session is deserialized into memory for the duration of a request, there shouldn't be any way for another user to affect it, nor even another request from the same user.

Yes, Steps 3 and 5 happen on in the same request! I agree the should be no way this would happen, our user code is simple in this regard, just writing and reading from the Session - we have been testing this and trying to work it out for about 3 days until we came here with this problem because of how unbelievable this is but so far we have found nothing in our user code that could be causing this.  We do read and write to the session using a simple SessionUtility that gets added by   services.AddTransient();
It is not easy to reproduce but we can do it, we can only reproduce this when the Crypto/OCSP bug happens that hangs up the thread for 15-20 seconds on our external staging servers - which we can do on Monday. I'm not able yet to reproduce this in our dev or internal test server setups.
It's a large app and a very small team so making a independent repro app would take a couple of days with a test rig, but I will create a gist today with our Startup.cs and all the relevant  code snippets and log snippets.
Otherwise my strategy today:

Add services.AddMvc();  after services.AddDistributedSqlServerCache(); in Startup.cs (could this really matter?)
Upgrade the app from 1.0.1 to 1.1

Ah, maybe its AddTransient config - looking into it now...

@Tratcher I think AddTransient is what we want but here is a gist with the relevant code and log Session log snippets from a case when this was reproduced - there are probably more than two people actually testing - this has gone into full external testing with a group of our customers.

One thing I noticed. Those three messages appear in the same seconds as when the mixup happens and its starts with key 8fd with count 1, then f8f with count 11, then again 8fd with count of 12, which is strange and could be a coincidence but why is second 8fd not 2 - it could seem that 8fd is picking up from count 11 from the other key.
2017-03-31 11:41:19.3001|4|Microsoft.AspNetCore.Session.DistributedSession|DEBUG|Session loaded; Key:8fd76c42-6cc8-e6aa-a54f-69756806a578, Id:9c3022ae-ebc8-a4b4-ef9e-9b501b03c05e, Count:1
2017-03-31 11:41:21.5308|5|Microsoft.AspNetCore.Session.DistributedSession|DEBUG|Session stored; Key:f8f50903-3c63-1ef9-adb5-03d2fc340de3, Id:a53f5a4f-bc3d-c2cc-0d16-41eab368d565, Count:11
2017-03-31 11:41:23.6835|5|Microsoft.AspNetCore.Session.DistributedSession|DEBUG|Session stored; Key:8fd76c42-6cc8-e6aa-a54f-69756806a578, Id:9c3022ae-ebc8-a4b4-ef9e-9b501b03c05e, Count:12

I see after reading the code that Count is really a count of the objects in the session so maybe this count action is normal.

We upgraded to 1.1.0 but we still managed to reproduce the problem.
Just before the mixup with user A we get the following session read from user B coming in from another company - after this user A has all the session information from user B:
2017-04-07 16:00:32.5866||Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector|TRACE|Performing unprotect operation to key {85d51abc-3f6f-4ebb-a3f7-204b91c1e646} with purposes ('osk2', 'SessionMiddleware').
2017-04-07 16:00:32.5866||Microsoft.AspNetCore.Session.DistributedSession|DEBUG|Session loaded; Key:b66c92ab-1bd2-a1fb-e317-ede5733ed0ec, Id:c86c915e-8732-c187-928f-d4c334ecd514, Count:14
Not sure if the "unprotected" is relevant there - we're looking into if the issue could be from the outside, for example something load balancer related or database related?
@Tratcher Any hints or ideas on what to test would be appreciated! :)

@Tratcher We have most likely found the bug, we will continue to test today and if it is not replicated I will close this ticket.

We had a AuthAttribute utility class that came over from the former .net framework version that defined a static attribute to temporarily hold the session interface - a pattern from .net framework. The session interface was passed in fresh on each call to the given function but a situation arose where both User A and User B were in the middle of the same function at the same time.
Our broken crypto function was/is actually making network calls in its 16-22 sec loop.
Kestrel seems to works async for network calls even if our user code is not async. This allowed user A to access to the static session of User B after User A came back from the async crypto network loop.

We fixed this by instantiating the AuthAttribute utility class so nothing is defined static.

Confirmed fixed :)


Audio not working on Acer Spin 11

Hello!
Audio isn't working for my device. I keep getting this when running the commands:
E: Unable to locate package firmware-sof-signed

Device is an Acer Chromebook Spin 11 (R751T / CP511), board name "electro".

Any estimate for when you'll have audio working?

It should already be working on Ubuntu. Follow the docs for audio.

lol, looks like you just recently addd that.
i built the installer before that was added, any way to download the script? (there probably definitely is i just don't know how)

nvm figured it out

the files does exist... but this happens

Run sudo apt install dos2unix, sudo dos2unix $(which apl-sof-setup-audio)

This is a reef device, and all of these devices are tested to work. See #6 for more details.

Hi, sorry for being quite late but still errors
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
gzip is already the newest version (1.10-4).
cgpt is already the newest version (0~R99-14469.B-1).
vboot-kernel-utils is already the newest version (0~R99-14469.B-1).
The following packages were automatically installed and are no longer required:
  hyphen-en-us javascript-common libpulsedsp libreoffice-help-common
  libreoffice-help-en-us libspeexdsp1 mythes-en-us node-normalize.css
  pulseaudio-utils
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package firmware-sof-signed
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package firmware-sof-signed

@MilkyDeveloper

@mrhappyma What distro are you on? Try Ubuntu.

Thanks for the quick reply @MilkyDeveloper, I am on Debian.
Is there any way to try troubleshooting on Debian? Building the usb took several hours (using pop os on a usb on a 9 year old mac).

Update:
As happened before, after the first restart after the audio script failing it now hangs on a blank screen with a white dash in the corner flashing when booting. I am re-building the USB with Ubuntu.


Mapped pursuit control

Now we have a 'mapped' path in the sim, we can make a control system to follow the path + adjust velocity accordingly.
sim_planning package
Photo from mapper script

Got an initial working commit https://github.com/QUT-Motorsport/QUTMS_Driverless/commit/898aaf6a4bd13f791f18354cf10541443fdac1dd
https://user-images.githubusercontent.com/43590806/154198070-897667c9-8e18-4137-8ea9-ed69ef4b500c.mp4


URGENT BUG MongoDB 3.2 An unknown error has occurred

I added in https://github.com/christkv/mongodb-core/blob/master/lib/topologies/server.js#L251 a console.error('err', err); and this was the actual error.  Otherwise it only tells me "An unknown error has occurred" it doesn't tell me "cannot read property id of undefined".  This is a definite error.  I'm on MongoDB v3.2.
[TypeError: Cannot read property 'id' of undefined]
error: An unknown error has occured, please try again response_time=43, path=/admin/stock, page=1, limit=10, , , status=302, response_type=text/html; charset=utf-8, type=api_error, message=An unknown error has occured, please try again, stack=MongoError: An unknown error has occured, please try again
    at null.messageHandler (/Users/foobar/Public/foobar/node_modules/mongoose/node_modules/mongodb/node_modules/mongodb-core/lib/topologies/server.js:246:29)
    at Socket.<anonymous> (/Users/foobar/Public/foobar/node_modules/mongoose/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js:262:22)
    at emitOne (events.js:77:13)
    at Socket.emit (events.js:169:7)
    at readableAddChunk (_stream_readable.js:146:16)
    at Socket.Readable.push (_stream_readable.js:110:10)
    at TCP.onread (net.js:523:20)

I am using a $lookup operation here, its erroring out as well.

I think it needs fixed like this:
-state.callbacks.flush(new MongoError(err));
if (_.isError(err))
  state.callbacks.flush(err);
else
  state.callbacks.flush(new MongoError(err));

This is because err in this try catch block can sometimes already be a MongoError error object per the following example:
{ [MongoError: TypeError: Cannot read property 'id' of undefined]
  name: 'MongoError',
  message: [TypeError: Cannot read property 'id' of undefined] }

Anyways, I'm still not sure why we're getting this "Cannot read property id of undefined" issue.

Sorry I don't know what version of the driver you are using. Would it be possible for you to upgrade to the latest and try again. I will have a look at your suggested fix once I get back from christmas vacation. Any more context provided would be helpful including how to reproduce.

I am seeing this error on 3.2 but not on 3.0.  The model that is throwing the error has a custom String _id.

@joshualo with what driver version and can you produce code and post it here that reproduces.

@christkv node driver 2.0.46.  I think I captured most the code.  Basically the syncWithExternal pulls a list of people (assume unique names), sets _id to name, updates or upserts them into db, also populating a cached list.  The init function is used to populate the cached list from scratch.  syncWithExternal works but we get the error when calling init.
Error:
MongoError: TypeError: Cannot read property 'id' of undefined

Code:
var schema = new mongoose.Schema({
  _id: {type: String, index: true},
  name: {type: String, required: true}
});

var personList = {};

schema.statics.initList = function(callback) {
  callback = callback || _.noop;

  this.find({}).exec((err, people) => {
    if (err) {
      console.log(err);
      return callback(err);
    }

    if (!people) {
      return callback(new Error());
    }

    for (var i = 0; i < people.length; i++) {
      var person = people[i];
      personList[person.name] = person;
    }

    callback();
  });
}

schema.statics.syncWithExternal = function(callback) {
  callback = callback || _.noop();

  var people = someExternalDataSource.toJSON();

  var update = (person, callback) => {
    if (!person.name) {
      return callback();
    }

    person._id = person.name;
    this.findByIdAndUpdate(person._id, person, {upsert: true, new: true}, function(err, person) {
      if (err) {
        return callback(err);
      }

      personList[person._id] = person;
    });
  }

  var finished = err => {
    return callback(err);
  }

  async.each(people, update, finished);
};

You might need to open a mongoose ticket for this one. But I would update mongoose first if you are on the 4.x branch first to get the latest node driver with official 3.2 support.

If id is missing it's because Mongoose 3.8 added it as a virtual and
you're using it with lean or something?
On Dec 29, 2015 6:34 PM, "Christian Amor Kvalheim"<EMAIL_ADDRESS>wrote:

You might need to open a mongoose ticket for this one. But I would update
mongoose first if you are on the 4.x branch first to get the latest node
driver with official 3.2 support.
—
Reply to this email directly or view it on GitHub
https://github.com/christkv/mongodb-core/issues/68#issuecomment-167899890
.

no further response, closing due to inactivity


Create minimal TUI with functioning tabs & CSS

Before submitting a pull request (PR), please read the contributing guide.
Description
What is this PR

[ ] Bug fix
[X] Addition of a new feature
[ ] Other

What does this PR do?
This PR adds a minimal TUI app built using the Textual package. While the window currently contains only 2 (largely empty) tabs, our intention is that this will ultimately form the skeleton of the DataShuttle graphical user interface.
References
Closes #185
How has this PR been tested?
At the current moment, the minimal TUI runs independently from the DataShuttle core software. Testing has been done by simply running the app via the terminal.
Is this a breaking change?
No.
Does this PR require an update to the documentation?
I don't think the app has reached a state where a change to documentation is warranted quite yet.
Checklist:

[X] The code has been tested locally
[ ] Tests have been added to cover all new functionality
[ ] The documentation has been updated to reflect any changes
[X] The code has been formatted with pre-commit

Hey @b-peri I wanted to play with this, so I added a couple of things to make it easier to install and run on linux/macOS. I'll let @JoeZiminski review this, but it's looking good!

Hey @b-peri I wanted to play with this, so I added a couple of things to make it easier to install and run on linux/macOS. I'll let @JoeZiminski review this, but it's looking good!

No problem @adamltyson, thanks! I've just put this in the issue (#185) as well but if you want to play around with the CSS bear in mind that Textual supports live CSS editing (See https://textual.textualize.io/guide/devtools/)

Thanks @b-peri! Looks great, we can merge this as-is into the new tui_dev_branch. I would suggest moving # Compose window into a docstring for the compose() function (+ a little more info), and removing the #Set path to TCSS and '# Set key-bindings comments as these are arguably self-evident from the code itself. However this is very minor / subjective and can be adressed on another PR. Looking good to go!

I forgot to mention for posterity, as discussed in #184 we will test at a more sensible time once a few more features are in place and we get user feedback.

Thanks @b-peri! Looks great, we can merge this as-is into the new tui_dev_branch. I would suggest moving # Compose window into a docstring for the compose() function (+ a little more info), and removing the #Set path to TCSS and '# Set key-bindings comments as these are arguably self-evident from the code itself. However this is very minor / subjective and can be adressed on another PR. Looking good to go!

Sounds good @JoeZiminski, thanks for the feedback! I've implemented your last few comments as part of #215, and so will close this PR now.


Pipeline Job to Automatically Build and Create Images for RaspberryPi4 Bullseye, RaspberryPi4 Bookworm, and amd64 Hybrid

Pipeline Job to Automatically Build and Create Images for RaspberryPi4 Bullseye, RaspberryPi4 Bookworm, and amd64 Hybrid
Description:
To align with the image release structure of LinuxCNC, we need to create a pipeline job that automatically builds and creates images for the following targets:

Raspberry Pi 4 Bullseye
Raspberry Pi 4 Bookworm
amd64 Hybrid

The images should mirror the structure and format used by LinuxCNC, ensuring that they are compatible with the current releases and users can seamlessly use these builds. This process should also account for creating and updating images for future versions.
Proposed Solution:

Create Pipeline Jobs:

Set up dedicated jobs in the CI pipeline to automate the build process for each of the target platforms.
The pipeline should cross-compile for ARM (for Raspberry Pi 4 images) and natively compile for amd64 (hybrid images).

Reference Existing LinuxCNC Build Structure:

Ensure that the images follow the structure currently provided by LinuxCNC:

Raspberry Pi Bullseye and Bookworm versions.
amd64 Hybrid builds for regular PCs.

This includes any necessary packaging, pre-installed dependencies (like real-time kernels), and configuration.

Utilize Existing Build Tools:

Leverage tools like linuxcnc-live-build to streamline the image creation process and ensure consistency with official LinuxCNC images.
Use QEMU for ARM emulation if needed to ensure image creation can be automated for Raspberry Pi 4 without direct access to ARM hardware.

Test and Validate Images:

Add automated testing to the pipeline to validate that the images boot correctly and that LinuxCNC runs as expected on each platform.
Perform basic configuration checks for PoKeys integration and the expected LinuxCNC configuration.

Store and Distribute the Images:

Store the generated images in the GitHub repository's releases or a suitable cloud storage platform.
Provide download links for users in the repository or related documentation.

Ensure Compatibility with the Latest LinuxCNC Versions:

Track and maintain compatibility with the latest stable releases of LinuxCNC, specifically for:

RaspberryPi 4 Bullseye
RaspberryPi 4 Bookworm
amd64 Hybrid.

Reference Images:

LinuxCNC 2.9.2 amd64 hybrid
LinuxCNC 2.9.3 amd64 hybrid
RaspberryPi4 Debian Bookworm 6.1.54 RT15 arm64

Tasks:

[ ] Create CI pipeline jobs for building Raspberry Pi 4 Bullseye, Bookworm, and amd64 hybrid images.
[ ] Align the structure of the images with existing LinuxCNC releases.
[ ] Use linuxcnc-live-build tools for packaging and compiling.
[ ] Automate testing and validation of images for booting and running LinuxCNC.
[ ] Provide download links for the generated images in the repository or cloud storage.

Acceptance Criteria:

[ ] Automated builds for Raspberry Pi 4 Bullseye, Bookworm, and amd64 hybrid are generated successfully.
[ ] Images match the structure of LinuxCNC releases and boot successfully.
[ ] PoKeys components and LinuxCNC integrations are tested and verified in each image.
[ ] Images are stored and distributed for download in the repository.

Reference Links:

Raspberry Pi OS
Cross-compilation for ARM on Raspberry Pi
LinuxCNC Building Documentation
LinuxCNC Downloads
QEMU for ARM Emulation
GitHub Actions with Raspberry Pi
LinuxCNC Live Build

update tests to just use the versions as provided on linuxcnc download page


RuntimeError: cusolver error: CUSOLVER_STATUS_INTERNAL_ERROR, when calling cusolverDnCreate(handle)

Hi, at present, I have deployed the environment and conducted inference testing according to the documentation (https://github.com/Yzichen/FlashOCC/blob/master/doc/install.md), but the following error message appears. Please help me to see if there is a problem with the model configuration.
WARNING:torch.distributed.run:

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

/home/users/yangchun.yan/venv/FlashOCC_venv/lib/python3.8/site-packages/mmdet/utils/setup_env.py:48: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
warnings.warn(
projects.mmdet3d_plugin
/home/users/yangchun.yan/venv/FlashOCC_venv/lib/python3.8/site-packages/mmdet/utils/setup_env.py:48: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
warnings.warn(
projects.mmdet3d_plugin
/home/users/yangchun.yan/venv/FlashOCC_venv/lib/python3.8/site-packages/mmdet/utils/setup_env.py:48: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
warnings.warn(
projects.mmdet3d_plugin
/home/users/yangchun.yan/venv/FlashOCC_venv/lib/python3.8/site-packages/mmdet/utils/setup_env.py:48: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
warnings.warn(
projects.mmdet3d_plugin
/home/users/yangchun.yan/venv/FlashOCC_venv/lib/python3.8/site-packages/mmdet/models/backbones/resnet.py:401: UserWarning: DeprecationWarning: pretrained is deprecated, please use "init_cfg" instead
warnings.warn('DeprecationWarning: pretrained is deprecated, '
/home/users/yangchun.yan/venv/FlashOCC_venv/lib/python3.8/site-packages/mmdet/models/backbones/resnet.py:401: UserWarning: DeprecationWarning: pretrained is deprecated, please use "init_cfg" instead
warnings.warn('DeprecationWarning: pretrained is deprecated, '
/home/users/yangchun.yan/venv/FlashOCC_venv/lib/python3.8/site-packages/mmdet/models/backbones/resnet.py:401: UserWarning: DeprecationWarning: pretrained is deprecated, please use "init_cfg" instead
warnings.warn('DeprecationWarning: pretrained is deprecated, '
/home/users/yangchun.yan/venv/FlashOCC_venv/lib/python3.8/site-packages/mmdet/models/backbones/resnet.py:401: UserWarning: DeprecationWarning: pretrained is deprecated, please use "init_cfg" instead
warnings.warn('DeprecationWarning: pretrained is deprecated, '
/home/users/yangchun.yan/venv/FlashOCC_venv/lib/python3.8/site-packages/mmdet/models/losses/cross_entropy_loss.py:239: UserWarning: Default avg_non_ignore is False, if you would like to ignore the certain label and average loss over non-ignore labels, which is the same with PyTorch official cross_entropy, set avg_non_ignore=True.
warnings.warn(
load checkpoint from local path: ckpts/flashocc-r50-256x704.pth
/home/users/yangchun.yan/venv/FlashOCC_venv/lib/python3.8/site-packages/mmdet/models/losses/cross_entropy_loss.py:239: UserWarning: Default avg_non_ignore is False, if you would like to ignore the certain label and average loss over non-ignore labels, which is the same with PyTorch official cross_entropy, set avg_non_ignore=True.
warnings.warn(
load checkpoint from local path: ckpts/flashocc-r50-256x704.pth
/home/users/yangchun.yan/venv/FlashOCC_venv/lib/python3.8/site-packages/mmdet/models/losses/cross_entropy_loss.py:239: UserWarning: Default avg_non_ignore is False, if you would like to ignore the certain label and average loss over non-ignore labels, which is the same with PyTorch official cross_entropy, set avg_non_ignore=True.
warnings.warn(
/home/users/yangchun.yan/venv/FlashOCC_venv/lib/python3.8/site-packages/mmdet/models/losses/cross_entropy_loss.py:239: UserWarning: Default avg_non_ignore is False, if you would like to ignore the certain label and average loss over non-ignore labels, which is the same with PyTorch official cross_entropy, set avg_non_ignore=True.
warnings.warn(
load checkpoint from local path: ckpts/flashocc-r50-256x704.pth
load checkpoint from local path: ckpts/flashocc-r50-256x704.pth
[                                                  ] 0/6019, elapsed: 0s, ETA:/home/users/yangchun.yan/FlashOCC/projects/mmdet3d_plugin/datasets/pipelines/loading.py:361: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
gt_boxes, gt_labels = torch.Tensor(gt_boxes), torch.tensor(gt_labels)
/home/users/yangchun.yan/FlashOCC/projects/mmdet3d_plugin/datasets/pipelines/loading.py:361: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
gt_boxes, gt_labels = torch.Tensor(gt_boxes), torch.tensor(gt_labels)
/home/users/yangchun.yan/FlashOCC/projects/mmdet3d_plugin/datasets/pipelines/loading.py:361: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
gt_boxes, gt_labels = torch.Tensor(gt_boxes), torch.tensor(gt_labels)
/home/users/yangchun.yan/FlashOCC/projects/mmdet3d_plugin/datasets/pipelines/loading.py:361: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
gt_boxes, gt_labels = torch.Tensor(gt_boxes), torch.tensor(gt_labels)
/home/users/yangchun.yan/FlashOCC/projects/mmdet3d_plugin/datasets/pipelines/loading.py:361: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
gt_boxes, gt_labels = torch.Tensor(gt_boxes), torch.tensor(gt_labels)
/home/users/yangchun.yan/FlashOCC/projects/mmdet3d_plugin/datasets/pipelines/loading.py:361: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
gt_boxes, gt_labels = torch.Tensor(gt_boxes), torch.tensor(gt_labels)
/home/users/yangchun.yan/FlashOCC/projects/mmdet3d_plugin/datasets/pipelines/loading.py:361: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
gt_boxes, gt_labels = torch.Tensor(gt_boxes), torch.tensor(gt_labels)
/home/users/yangchun.yan/FlashOCC/projects/mmdet3d_plugin/datasets/pipelines/loading.py:361: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
gt_boxes, gt_labels = torch.Tensor(gt_boxes), torch.tensor(gt_labels)
/home/users/yangchun.yan/FlashOCC/projects/mmdet3d_plugin/datasets/pipelines/loading.py:361: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
gt_boxes, gt_labels = torch.Tensor(gt_boxes), torch.tensor(gt_labels)
/home/users/yangchun.yan/FlashOCC/projects/mmdet3d_plugin/datasets/pipelines/loading.py:361: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
gt_boxes, gt_labels = torch.Tensor(gt_boxes), torch.tensor(gt_labels)
/home/users/yangchun.yan/FlashOCC/projects/mmdet3d_plugin/datasets/pipelines/loading.py:361: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
gt_boxes, gt_labels = torch.Tensor(gt_boxes), torch.tensor(gt_labels)
/home/users/yangchun.yan/FlashOCC/projects/mmdet3d_plugin/datasets/pipelines/loading.py:361: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
gt_boxes, gt_labels = torch.Tensor(gt_boxes), torch.tensor(gt_labels)
/home/users/yangchun.yan/FlashOCC/projects/mmdet3d_plugin/datasets/pipelines/loading.py:361: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
gt_boxes, gt_labels = torch.Tensor(gt_boxes), torch.tensor(gt_labels)
/home/users/yangchun.yan/FlashOCC/projects/mmdet3d_plugin/datasets/pipelines/loading.py:361: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
gt_boxes, gt_labels = torch.Tensor(gt_boxes), torch.tensor(gt_labels)
/home/users/yangchun.yan/FlashOCC/projects/mmdet3d_plugin/datasets/pipelines/loading.py:361: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
gt_boxes, gt_labels = torch.Tensor(gt_boxes), torch.tensor(gt_labels)
/home/users/yangchun.yan/FlashOCC/projects/mmdet3d_plugin/datasets/pipelines/loading.py:361: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
gt_boxes, gt_labels = torch.Tensor(gt_boxes), torch.tensor(gt_labels)
Traceback (most recent call last):
File "tools/test.py", line 290, in
main()
File "tools/test.py", line 266, in main
outputs = multi_gpu_test(model, data_loader, args.tmpdir,
File "/home/users/yangchun.yan/venv/FlashOCC_venv/lib/python3.8/site-packages/mmdet/apis/test.py", line 109, in multi_gpu_test
result = model(return_loss=False, rescale=True, **data)
File "/home/users/yangchun.yan/venv/FlashOCC_venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/users/yangchun.yan/venv/FlashOCC_venv/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 886, in forward
output = self.module(*inputs[0], **kwargs[0])
File "/home/users/yangchun.yan/venv/FlashOCC_venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/users/yangchun.yan/venv/FlashOCC_venv/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 116, in new_func
return old_func(*args, **kwargs)
File "/home/users/yangchun.yan/FlashOCC/mmdetection3d/mmdet3d/models/detectors/base.py", line 62, in forward
return self.forward_test(**kwargs)
File "/home/users/yangchun.yan/FlashOCC/projects/mmdet3d_plugin/models/detectors/bevdet.py", line 201, in forward_test
return self.simple_test(points[0], img_metas[0], img_inputs[0],
File "/home/users/yangchun.yan/FlashOCC/projects/mmdet3d_plugin/models/detectors/bevdet_occ.py", line 111, in simple_test
img_feats, _, _ = self.extract_feat(
File "/home/users/yangchun.yan/FlashOCC/projects/mmdet3d_plugin/models/detectors/bevdet.py", line 116, in extract_feat
img_feats, depth = self.extract_img_feat(img_inputs, img_metas, **kwargs)
File "/home/users/yangchun.yan/FlashOCC/projects/mmdet3d_plugin/models/detectors/bevdet.py", line 94, in extract_img_feat
img_inputs = self.prepare_inputs(img_inputs)
File "/home/users/yangchun.yan/FlashOCC/projects/mmdet3d_plugin/models/detectors/bevdet.py", line 72, in prepare_inputs
global2keyego = torch.inverse(keyego2global.double())   # (B, 1, 4, 4)
RuntimeError: cusolver error: CUSOLVER_STATUS_INTERNAL_ERROR, when calling cusolverDnCreate(handle)

Since I deployed on 4090gpu, the above error occurred and changed to 3090 to pass.

so，how to run on 4090cpu


[Feature] Add status messages of a search while search is ongoing

While searching for a Trade route, add custom messages of the status in the main progress bar and sub-progress bar.
For main_progress bar, instead of showing 85% you could show :
85% - Searching for destination commodities

Use QProgressDialog ?


nonadmin should be non_admin

There is an error in the python example. The elevation level should be non_admin instead of nonadmin: https://docs.microsoft.com/en-us/python/api/azure-batch/azure.batch.models.elevationlevel?view=azure-python

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: d7cb180a-6147-7108-c866-c2f3ca552f52
Version Independent ID: 446a19b1-c135-2ad7-1b4e-f1d4dabfd0cd
Content: Run tasks under user accounts - Azure Batch
Content Source: articles/batch/batch-user-accounts.md
Service: batch
GitHub Login: @laurenhughes
Microsoft Alias: lahugh

Thanks for the feedback! We are currently investigating and will update you shortly.

@mkiernan I will assign this to the content author to review as I am not overly familiar with Python.
@laurenhughes can you confirm? I can always update the doc if needed.

@mkiernan Thanks for pointing this out! The doc has been updated and will be published within the next few hours.
#please-close


Attention net classification

Hi,
This is for the coding assessment Slava asked me to do
(build a simple machine learning model and submit your code for review on our open-source platform:
https://produvia.com/ai-platform ).
Thanks,
Amir

@amirfaraji Well done. Please integrate MLflow into your pull request. MLflow is a code dependency at ai-platform. Our README file contains more information about this.

Hi @lucky-singh,
I believe I was able to address the issues you mentioned in your review. Parameters, metrics and models are all logged with mlflow. I provided more details about the model, files, parameters, usage and examples in the README file. Finally, the dataset, model and pretrained-weights are downloaded on runtime.

@mlaradji please review this contribution


K8SPS-97: Add delete-backup finalizer

https://jira.percona.com/browse/K8SPS-97

Test name
Status

demand-backup
passed

auto-config
passed

init-deploy
passed

gr-init-deploy
passed

scaling
passed

monitoring
passed

config
passed

users
passed

semi-sync
passed

service-per-pod
passed

sidecars
passed

limits
passed

We run 12 out of 12

commit: https://github.com/percona/percona-server-mysql-operator/pull/158/commits/2490d4a9837d0b51d045e5bad429f57cbf8bf438
image: perconalab/percona-server-mysql-operator:PR-158-2490d4a

Test name
Status

demand-backup
passed

init-deploy
passed

auto-config
passed

gr-init-deploy
passed

monitoring
passed

semi-sync
passed

scaling
passed

config
passed

service-per-pod
passed

users
passed

sidecars
passed

limits
passed

We run 12 out of 12

commit: https://github.com/percona/percona-server-mysql-operator/pull/158/commits/5e2c2548f20740dae790f3f131bb99ed0edfed5f
image: perconalab/percona-server-mysql-operator:PR-158-5e2c254

After backup I have these in my bucket:
* cluster1-2022-09-13-10:13:06-full.md5

* cluster1-2022-09-13-10:13:06-full/

After backup delete I have this:
* cluster1-2022-09-13-10:13:06-full.md5

Should .md5 files be deleted as well?

md5 files should be deleted, but problem is on the xbcloud side

After backup I have these in my bucket:
* cluster1-2022-09-13-10:13:06-full.md5

* cluster1-2022-09-13-10:13:06-full/

After backup delete I have this:
* cluster1-2022-09-13-10:13:06-full.md5

Should .md5 files be deleted as well?

md5 files should be deleted, but problem is on the xbcloud side PXB-1854

Aha, got it.

@pooknull did you maybe managed to reproduce the errors I posted earlier?

@inelpandzic I reproduced the error. Here is the fix https://github.com/percona/percona-server-mysql-operator/pull/158/commits/2f0ec1c09eec8b85189b2f7cb394e0bf72b8a712

Test name
Status

auto-config
passed

demand-backup
failed

init-deploy
passed

gr-init-deploy
passed

monitoring
passed

semi-sync
passed

service-per-pod
passed

config
passed

scaling
passed

sidecars
passed

limits
passed

users
passed

We run 12 out of 12

commit: https://github.com/percona/percona-server-mysql-operator/pull/158/commits/562c994bf21e70846fcfd75f588f3f4cb0d218f6
image: perconalab/percona-server-mysql-operator:PR-158-562c994

Test name
Status

demand-backup
failed

auto-config
passed

init-deploy
passed

gr-init-deploy
passed

monitoring
passed

semi-sync
passed

service-per-pod
passed

scaling
passed

sidecars
passed

limits
passed

users
passed

config
passed

We run 12 out of 12

commit: https://github.com/percona/percona-server-mysql-operator/pull/158/commits/fc05cf465363a3e15c51d8268ececca5b5cf581a
image: perconalab/percona-server-mysql-operator:PR-158-fc05cf4

Test name
Status

auto-config
passed

config
passed

demand-backup
failed

init-deploy
passed

gr-init-deploy
passed

monitoring
passed

semi-sync
passed

service-per-pod
passed

sidecars
passed

limits
passed

scaling
passed

users
passed

We run 12 out of 12

commit: https://github.com/percona/percona-server-mysql-operator/pull/158/commits/fc05cf465363a3e15c51d8268ececca5b5cf581a
image: perconalab/percona-server-mysql-operator:PR-158-fc05cf4

Test name
Status

auto-config
passed

demand-backup
passed

init-deploy
passed

gr-init-deploy
passed

monitoring
passed

config
passed

semi-sync
passed

service-per-pod
passed

sidecars
passed

limits
passed

scaling
passed

users
passed

We run 12 out of 12

commit: https://github.com/percona/percona-server-mysql-operator/pull/158/commits/05fbd1800e1d96878c9d8841842fe1320a450443
image: perconalab/percona-server-mysql-operator:PR-158-05fbd18

Test name
Status

auto-config
passed

demand-backup
failed

init-deploy
passed

gr-init-deploy
passed

monitoring
passed

semi-sync
passed

service-per-pod
passed

config
passed

sidecars
passed

limits
passed

scaling
passed

users
passed

We run 12 out of 12

commit: https://github.com/percona/percona-server-mysql-operator/pull/158/commits/c5333d99b8176fd5a69c8fdb6b927ae0f585f5b3
image: perconalab/percona-server-mysql-operator:PR-158-c5333d9

Test name
Status

demand-backup
passed

auto-config
passed

init-deploy
passed

gr-init-deploy
passed

monitoring
passed

semi-sync
passed

service-per-pod
passed

scaling
passed

config
passed

sidecars
passed

limits
passed

users
passed

We run 12 out of 12

commit: https://github.com/percona/percona-server-mysql-operator/pull/158/commits/8894c00da2acfc5a55eef9ab510908e8601b60db
image: perconalab/percona-server-mysql-operator:PR-158-8894c00


Fix the exception of show tables and use db

Fixes #I1006.
Changes proposed in this pull request:

Moidfy addMemoryQueryResultRows.
Modify appendSchemaPlaceholder

Codecov Report

Merging #1013 into dev will decrease coverage by 0.01%.
The diff coverage is 60%.

@@            Coverage Diff             @@
##              dev    #1013      +/-   ##
==========================================
- Coverage   54.73%   54.72%   -0.02%
==========================================
  Files         634      634
  Lines       12357    12360       +3
  Branches     1945     1946       +1
==========================================
  Hits         6764     6764
- Misses       5181     5187       +6
+ Partials      412      409       -3

Impacted Files
Coverage Δ

...ava/io/shardingsphere/core/rewrite/SQLBuilder.java
85.29% <50%> (-1.07%)
:arrow_down:

...e/core/merger/dal/show/ShowTablesMergedResult.java
90.9% <66.66%> (+3.4%)
:arrow_up:

...al/state/datasource/DataSourceListenerManager.java
36.66% <0%> (-3.34%)
:arrow_down:

...ternal/state/instance/InstanceListenerManager.java
29.41% <0%> (-2.95%)
:arrow_down:

.../internal/config/ConfigurationListenerManager.java
54.05% <0%> (-2.71%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7a84aaa...e1216b9. Read the comment docs.


satcluster02 docker fehler: ... docker.sock: read: connection reset by peer

some docker commands are not working anymore after satcluster02 failure
e.g. docker ps => "An error occurred trying to connect: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json: read unix @->/var/run/docker.sock: read: connection reset by peer"
restart of vm doesnt help

deleted all docker images, containers, etc:
sudo rm -rf /var/lib/docker
docker restart
docker ps working again
restart live with jenkins

reinstalled docker on satcluster02:
sudo apt-get install docker-engine
file is there again: /var/lib/docker/network/files/local-kv.db

This problem occured again, it could be fixed by deleting the following folders:

/var/lib/docker/image, /var/lib/docker/containers, /var/lib/docker/aufs, /var/lib/docker/tmp
still not clear what the problem really was, maybe the hard drive was full? test this next time before ...


HWKAGENT-62 put agent subsystem in WF add-ons location

@jmazzitelli could you please review and eventually merge?

ce2455d makes Checkstyle happy

- configurationBldr.modulesHome("modules");
+ configurationBldr.modulesHome("modules/system/add-ons/hawkular-agent");

I am glad this looks to have been a simple one-line fix. I was crossing my fingers it didn't require more than that. :)
I assume you ran the installer to see that it went in the proper place and then ran the actual WF instance to see that the agent really does deploy and start up properly when placed in this add-ons directory?

I am glad this looks to have been a simple one-line fix

I also changed the location of the module.xml in the source tree.

I assume you ran the installer

No, I have not. Don't we have an itest checking this?

No, I have not. Don't we have an itest checking this?

Even if we have an itest, its always a good idea to always just run the software. As for this, I don't know if an itest actually runs the installer and then runs the WF instance with that installed agent. What WILL test this is the build of hawkular distro - because the hawkular distro build uses the installer to install the agent. So if you build hawkular distro and run it, if you see the agent starting up, this works.

OK, do not merge yet, I'll add such a test. I should actually also add a simple test checking that the Agent is writing to Metrics, because I suspect that the writes are not successful ATM, but the build is passing anyway.

911f952 brings some integration tests to ensure that Agent installer is able to:

Install on top of a plain WildFly
that the WF with the freshly installed agent boots and writes data to Inventory and Metrics hosted on an external Hawkular instance

ATM, the named tests live in a separate Maven module hawkular-wildfly-agent-installer-itest. I started like that to be able to work in isolation from the tests available at present (to exclude interferences) but I am not particularly happy with the result, because there are as many as three container instances stared during the itests: once plain WF with Agent installed and two Hawkular instance. I think that it would be better to merge hawkular-wildfly-agent-installer-itest with the existing hawkular-wildfly-agent-itest so that there is just the plain WF and one Hawkular. However, if Travis is able to swallow the setup like this, I vote for merging 911f952 and eventually improve later so that this does not block Hawkular M11. WDYT, @jmazzitelli ?

I totally forget the agent extension jar is packaged with the module subdirectories. So no code changes needed to be made, I guess - just package up the assembly so the jar has the 'add-ons' and that's all that was needed. Since travis is green, I'm good to merge this.

I'm good to merge this.

Could me or you release the Agent for M11 after merging?

@ppalaga if you want to merge/release now, go ahead. I have nothing that needs to go into M11, so it can be done now or tomorrow, it doesn't matter. I can release if you can't, but I would have to do it tomorrow.


[Bug]: Shared context doesn't seem to work with runtime module federation[Bug]:

System Info
Need to install the following packages:
<EMAIL_ADDRESS>Ok to proceed? (y) y
System:
OS: Linux 6.1 Manjaro Linux
CPU: (12) x64 Intel(R) Core(TM) i7-9850H CPU @ 2.60GHz
Memory: 13.32 GB / 30.96 GB
Container: Yes
Shell: 3.7.1 - /usr/bin/fish
Binaries:
Node: 18.19.0 - /run/user/1000/fnm_multishells/305667_1720381529332/bin/node
Yarn: 1.22.22 - /run/user/1000/fnm_multishells/305667_1720381529332/bin/yarn
npm: 10.2.3 - /run/user/1000/fnm_multishells/305667_1720381529332/bin/npm
npmPackages:
@rspack/cli: ^1.0.0-alpha.0 => 1.0.0-alpha.0
Details
I am trying to share mantine with module federation. However, mantine requires a context to be created. Even though I have specified @mantine/core as a singleton in both the remote and shell. It still says that the context has not been initialized and the app crashes.
Reproduce link
https://github.com/AdamJSoftware/rspack-module-federation-bug
Reproduce Steps

cd remote
yarn install
yarn dev
cd shell
yarn install
yarn dev
Navigate to http://localhost:3000 and you will see

react.development.js:1618 Uncaught TypeError: Cannot read properties of null (reading 'useContext')
    at useContext (react.development.js:1618:1)
    at useMantineTheme (MantineThemeProvider.mjs:10:1)
    at useProps (use-props.mjs:9:1)
    at Text.mjs:46:1
    at renderWithHooks (react-dom.development.js:15486:1)
    at updateForwardRef (react-dom.development.js:19245:1)
    at beginWork (react-dom.development.js:21675:1)
    at HTMLUnknownElement.callCallback (react-dom.development.js:4164:1)
    at Object.invokeGuardedCallbackDev (react-dom.development.js:4213:1)
    at invokeGuardedCallback (react-dom.development.js:4277:1)

Similar issue is happening here: https://github.com/vercel/next.js/discussions/64887
Could this be an issue with the module federation spec itself?


ePub Library not support the version of iOS 13 and perfectly running in iOS 12.4.2. Please help me to find the solution

General Instructions

Please use a short, but meaningful title.
Delete everything in italics before submitting issue
Make sure to fill in as much information as possible

This issue is a Bug|Feature Request|Enhancement|Question|Task|Other
(choose ONE from the above list and delete the others)
Related issue(s) and/or pull request(s)
Example: See also #NUMBER
Expected Behaviour
What you expected Readium to do...
Observed behaviour
Instead, Readium does...
Steps to reproduce

First, do this
Then do this
etc.

Test file(s)
If relevant, please indicate public EPUB URL(s), or whether you can share ebook(s) privately.  It is often next to impossible to reproduce bugs without the original file.
Product
Choose one of the following (and delete the others)

Readium Chrome extension

latest official version available from the Chrome Web Store
Version of the Chrome browser
OS and version

Readium cloud reader app

latest development build uploaded at https://readium.firebaseapp.com OR a custom integration (list public deployment URL, if any)
Browser and version
OS and version

Native application (Readium SDK C++)

official "launcher" sample OR propietary app
OS and version

Hybrid app (ReadiumJS)

Electron OR Apache Cordova OR ?
OS and version

Other?

Additional information
Please list any additional information that might help us

Closing for now as it is stale and there isn't any real info.