Use volatile read in BaseRevision::is_supported()

I'm making a kernel using the limine-rs template as reference and noticed that the _start() function included in the template doesn't draw anything on the framebuffer:
#[no_mangle]
unsafe extern "C" fn _start() -> ! {
    assert!(BASE_REVISION.is_supported());

    if let Some(framebuffer_response) = FRAMEBUFFER_REQUEST.get_response() {
        if let Some(framebuffer) = framebuffer_response.framebuffers().next() {
            for i in 0..100_u64 {
                // Calculate the pixel offset using the framebuffer information we obtained above.
                // We skip `i` scanlines (pitch is provided in bytes) and add `i * 4` to skip `i` pixels forward.
                let pixel_offset = i * framebuffer.pitch() + i * 4;

                // Write 0xFFFFFFFF to the provided pixel offset to fill it white.
                *(framebuffer.addr().add(pixel_offset as usize) as *mut u32) = 0xFFFFFFFF;
            }
        }
    }

    hcf();
}

Looking at the disassembly:
Disassembly of section .text:

ffffffff80000000 <_start>:
ffffffff80000000:       50                      push   rax
ffffffff80000001:       48 8d 3d f0 10 00 00    lea    rdi,[rip+0x10f0]        # ffffffff800010f8 <_ZN36_$LT$T$u20$as$u20$core..any..Any$GT$7type_id17h654c8c39a6067c34E+0x1018>
ffffffff80000008:       48 8d 15 39 21 00 00    lea    rdx,[rip+0x2139]        # ffffffff80002148 <_ZN36_$LT$T$u20$as$u20$core..any..Any$GT$7type_id17h654c8c39a6067c34E+0x2068>
ffffffff8000000f:       be 2e 00 00 00          mov    esi,0x2e
ffffffff80000014:       e8 77 00 00 00          call   ffffffff80000090 <_ZN4core9panicking5panic17h951cafdfee7cec9cE>
ffffffff80000019:       cc                      int3
ffffffff8000001a:       cc                      int3
ffffffff8000001b:       cc                      int3
ffffffff8000001c:       cc                      int3
ffffffff8000001d:       cc                      int3
ffffffff8000001e:       cc                      int3
ffffffff8000001f:       cc                      int3

it seems that _start() just immediately panics because the compiler thinks that assert!(BASE_REVISION.is_supported()) will always fail. Removing the assertion fixes this.
The problem seems to be that BaseRevision::is_supported() doesn't use a volatile read:
/// Check whether the given revision is supported by the bootloader.
pub fn is_supported(&self) -> bool {
    (unsafe { *self.revision.get() }) == 0 // Should use read_volatile() instead of *
}

and so the compiler assumes that the read will always return 1, because that's whats written in BaseRevision::new(). Making it a volatile read fixes this.

I'll fix this in the next release (probably later today). Thanks for the report.

@UditDey is your BASE_REVISION static public?
read_volatile is not correct here. It is only for use on memory allocated outside of the Rust abstract machine. This normal read should just work, as long as the static is marked public (and potentially no_mangle). If your static is already public, this is probably a rustc bug.

Well, read_volatile is for memory that is read/written outside the abstract machine, not necessarily allocated outside it.
If I've understood correctly, the base revision tag is modified by the bootloader to communicate whether or not the requested revision is supported. And that's what BaseRevision::is_supported() seems to be trying to do, read back that field after we've booted to see whether 0 has been written there by the bootloader or not. This is a write from outside the rust abstract machine and absolutely warrants a volatile read.
Even the other tags use volatile reads to get their response. Look at Response<T>. This is why the other tags work without making them public/#[no_mangle]
But I also experimented with making it public/#[no_mangle] and it worked without the volatile read, which is interesting. Maybe #[no_mangle] makes rust relax it's assumptions?
In any case, making it a volatile read both removes the need to make it public/#[no_mangle] and also makes it consistent with how the other responses are read.

Yes, the others are actually mistaken in using the volatile read. Not making it public is actually UB.

That is surprising, how does that count as UB? I'm curious

Unless the static is public/no_mangle, it is considered internal, and assumed to not be modified outside of local code. We can prove that this is UB, because it has caused this miscompilation. A miscompilation can only occur if UB is involved (excluding a compiler bug). Therefore, some UB must have occured. Obviously, the read itself is not UB, because read only requires that the pointer is valid and the data satisfies the type's validity invariant. Modifying an UnsafeCell is not UB, because that is the purpose of an UnsafeCell. Therefore, the UB must have been due to the static not being public. Indeed, the miscompilation was caused by the compiler's assumption that because the static was internal, it could not be externally modified.

I'm sorry but I think you're completely incorrect. There was no "miscompilation", I don't see any UB here. The compiler elided away the pointer read because we didn't tell it that the bootloader can swoop in and modify the original value, and the way you tell it that is by using a volatile read.
If you have an internal value that can be modified by something external, you use a volatile read. That is true in C, it is true in rust.
And although it seems that public/'#[no_mangle]' has the same effect, I'm not aware of this property being documented anywhere. If you want to do it correctly, and make sure it stays correct in all future compiler versions, you use a volatile read.
I'm surprised you're confused about this considering that you got this right in Response<T> where you do in fact use a volatile read! I have never in my life seen marking something as public/#[no_mangle] be touted as equivalent to marking it as volatile.

There was by definition a miscompilation. This code should read the value. The fact that it was elided means that either the compiler did something wrong, or you did not uphold your promises to it.
The property of #[no_mangle] fixing this issue is related to how it interacts with linkage. Without #[no_mangle], the static is marked as a local symbol. It is only marked as exported with #[no_mangle]. See this discussion as well as the ABI for details on that. This is not equivalent to volatile. It is a requirement to safely write to the value from outside.
I will admit that this is not a very well-walked path. It is not written in black and white. However, no_mangle is intended to work this way. When I initially wrote Response, this didn't occur to me at all. I didn't question how a bug like this could happen without UB. But now it is clear that there is UB involved.

There was not a miscompilation. Since BASE_REVISION is not referenced anywhere else in the program, the compiler is allowed to remove it. To force the compiler to keep it in the final binary, the #[used] attribute can be used (see here), and it must be used on all limine requests. I've opened limine-bootloader/limine-rust-template#8 to fix this.

Perhaps miscompilation was the wrong word on my part. I'll look over your PR.

I objdump | grep'd the generated elf and here's what I saw:

Without #[used] the BASE_REVISION static is removed entirely, and the pointer read in is_supported() is elided
#[no_mangle] also works since it implies #[keep]
read_volatile() also makes sure the static isn't removed

So, technically only one of these is necessary. However this isn't guaranteed to remain true since, technically, #[used] can stop the static from being removed, but not stop the pointer read from being elided.
In fact, read this section from the RFC that introduced #[used]:

Yet another workaround is to force a volatile load of the vector table in some part of the program, usually before main. The compiler will always keep the vector table in this case but this alternative incurs in the cost of a load operation that will never be optimized away by the compiler.

This section explains why read_volatile() has the same effect, and also implies that #[used] is not the same as read_volatile() and therefore if we just use #[used] its technically possible for the static to be kept around but for the pointer read to still be elided.
So basically, I think the best option is to use both #[used] and read_volatile(), the first ensures that the static isn't removed, and the second ensures that the pointer read is not optimized away

Actually, @UditDey, can you verify whether or not the symbol is considered public? Afaik only the no_mangle will be, but I'm not sure.

If by public you mean exported and visible in the symbol table, then no. It won't be public unless #[no_mangle] or #[export] are used. But the symbol doesn't need to be public. In case you're confused about it, the bootloader finds the tags by scanning the file for magic numbers, it doesn't refer to the symbol table at all.

Yes, I know that. But if it's not public, llvm still has every right to elide it, and used is not perfect.

After rereading the rfc, it seems that #[used] appends the static to llvm's @llvm.used list. This makes the compiler behave as if there is a reference to the static that it cannot see (effectively making it public), and it will thus keep it in the binary. Since limine requests have interior mutability via UnsafeCell, the compiler should also not be able to elide the pointer read.
However, it seems that #[used] does not prevent the linker from removing the static if it is not used by any function. I've updated the PR with a comment explaining this.

Alright, then #[used] is completely fine. I'll accept your PR.
As for this issue, I'm not sure how we should proceed. The technically correct solution would be to remove all volatile reads, but the one which would cause fewer bugs would be to add a volatile read in BaseRevision. Let me know if you have a preference.

Since limine requests have interior mutability via UnsafeCell, the compiler should also not be able to elide the pointer read

This is false. UnsafeCell does not ensure that reads to it are never elided. It's only purpose is to enable interior mutability.
Just to make my point, compare the output assembly on this and this. The one with read_volatile() actually does the read, whereas the one without it elides it, despite both using UnsafeCell. It is absolutely not a rule of the language that a pointer originating from an UnsafeCell behaves any different than any other pointer. If you want to prevent the read elision, you use read_volatile().

The technically correct solution would be to remove all volatile reads

Again, I don't understand where this is coming from. Unless the rust docs actually say that using #[keep] ensures no pointer read elision, the only fool proof way to prevent that, in every future rust compiler version, is to use what is actually documented and guaranteed to work: read_volatile()
And even if you're fine with relying on undocumented behavior, how do you conclude the correct solution is to remove whatever volatile reads there are??
I am upset because what I thought was going to be a one-line fix has turned into a discussion of the semantics of compiler optimizations :(
I'll reiterate: The best solution is to both  use #[keep] on all our request statics, and read_volatile() for every response read

The reason why the compiler was able to elide the pointer read in your example was because the compiler could (correctly) infer that the value could not have changed by the time it was read. However, marking a static with #[used] forces the compiler/llvm to assume some other unknown part of the code references it. However, if the static could not change, the compiler could still elide it (see here). But since UnsafeCell provides interior mutability, the compiler can no longer make that assumption.
Imo, volatile reads are not necessary and could prevent correct optimizations such as reordering reads. The need to use #[used] should be documented though.

I don't imagine checking the base revision of the boot protocol is a particularly time-sensitive operation. Volatile reads seems more practical for end users than #[used], and besides, Limine already expects you to use volatile in C.

Alright, I suppose I'll just add the volatile then. I'd still like that PR to land, however.

However, marking a static with #[used] forces the compiler/llvm to assume some other unknown part of the code references it

If you can find the part of the rust docs that states that #[used] suppresses pointer optimizations I'll agree right away. However it's stated purpose is to just ensure that the item is not removed. The fact that it also suppresses pointer optimization is incidental, undocumented behavior that is not guaranteed to remain.
Look, I'm not being cranky for no reason here. I've been bitten by things like this!! I've written low level rust code that worked fine, I run rustup update and it stops working, because turns out that all along I was relying on incidental behavior instead of following the stated rules.
And as @TobiasSN points out, you literally need the volatile marker in C too!! Would you insist that instead of making it volatile you should __declspec(export) it?
And yes @lylythechosenone, that PR should land. #[keep] is a good idea (and so is read_volatile() :P)

If you can find the part of the rust docs that states that #[used] suppresses pointer optimizations I'll agree right away.

While there is nothing that explicitly states this behavior, it follows logically from the given conditions. According to the rfc,

The #[used] attribute can only be used on static variables. Static variables marked with this attribute will be appended to the special @llvm.used global variable when lowered to LLVM IR.

This tells rust/llvm that the static is reachable from some unknown outside code (by limine in this case). Also, the static is mutable via UnsafeCell. Since the unknown code can mutate the static, it follows that the compiler will not be able to make any assumptions about its value, preventing inlining. To me, it seems that this behavior is not incidental; rather, it follows logically from the condition that it can be mutated by unknown code.

And as @TobiasSN points out, you literally need the volatile marker in C too!! Would you insist that instead of making it volatile you should __declspec(export) it?

In the limine C template, removing volatile from
static volatile LIMINE_BASE_REVISION(1);

does indeed break the example. However, replacing it with __attribute((used)) (the C equivalent of #[used]) fixes it. Perhaps the C example could be updated to use this in the future.

I don't imagine checking the base revision of the boot protocol is a particularly performance-sensitive operation.

I agree, but there is a significant difference in generated code when the read is able to be hoisted out of a loop (https://godbolt.org/z/7xE6PjGM7). Granted, this should occur rarely, if at all, in real code.
In my opinion, the biggest reason for volatile operations is that it would mitigate the effects of someone forgetting to mark statics with #[used]. However, I think that documenting why limine requests need to be marked with #[used] along with being referenced in a function should prevent this from happening. Perhaps the limine utils could eventually have the capability to warn against a missing base revision tag.

Perhaps the limine utils could eventually have the capability to warn against a missing base revision tag.

This is a decent idea. I'll bring it up to mint.


INT-B-18881 Add Search SC

Agility ticket
Summary
Adding search capability to the Services Counselors move queue.
How to test

Access the Office app
Login as a Services Counselor
Click the Search tab
Search for a move
Click a move

Frontend

[ ] There are no aXe warnings for UI.
[ ] This works in Supported Browsers and their phone views (Chrome, Firefox, Edge).
[ ] There are no new console errors in the browser devtools.
[ ] There are no new console errors in the test output.
[ ] If this PR adds a new component to Storybook, it ensures the component is fully responsive, OR if it is intentionally not, a wrapping div using the officeApp class or custom min-width styling is used to hide any states the would not be visible to the user.
[ ] This change meets the standards for Section 508 compliance.

Screenshots

Warnings

:warning:
This PR does not include changes to unit tests, even though it affects app code.

  Generated by :no_entry_sign: dangerJS against 51a62b01f0c13f80ff83758b1b413c0a26b95bd4

Requirements are being revisited, closing until they are clear.


support micro-profile-sureness examples

support example use sureness protect microprofile- https://start.microprofile.io/

LGTM thanks.


added possibility for specifiying a path relative to the filesystem a…

…dapter
I am using flysystem for various file operations. With this, you can define a base filesystem and load the twig templates from a subdirectory.
I'm sure, there are more elegant ways to do this, feel free to criticise or enhance my edit.

Hi Marcus!
Thanks a bunch for your contribution!
I'd be happy to merge once travis is happy, please see https://travis-ci.org/cedricziel/twig-loader-flysystem/jobs/155084874#L313 - it refers to declaring two properties on the same line as I commented.
Also: I know the loader is pretty small - but I feel like we could concatenate the string once instead of three times, as I commented - what do you think?

Moin Cedric,
I changed the declaration of the properties.
Feel free to change the concentration, this was only a quick hack for me, I'm deep in my project (hcsfng-frontend) right now, just wanted to make it work for me. With the next push I will include my fork / your merged package.

My pleasure! Thanks for supplying this package!

FYI: I just tagged v1.1.0 and it's live on packagist

Thanks a lot.
Btw. I forgot to add the annotation for the new argument in the constructor. :/

Fixed :)


⚠ Bofrost.at has degraded performance

In 3754a98, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 2147 ms

Resolved: Bofrost.at performance has improved in 29367e0 after 6 minutes.


Change snap, localBounds and boundsSnap to const declaration at Manipulate.

Those arguments are not modified.
I think  'const float*' better than 'float*'.

Thank you!


Lexing comments

Hi,
nice project and tutorial series!
I have a question regarding your 14th part of the tutorial: Is there any future benefit to have an explicit comment token? Without doc comments etc. in mind it would be easier to filter comments with a simple logos::skip like:
pub(crate) enum SyntaxKind {
// snip

    #[error]
    #[regex("#.*"), logos::skip)] // added here because Error is some kind of fallback token
    Error,

// snip
}

Less code, easier to understand, possibly faster and a simpler parser tree. The only drawback is that it is not possible to parse the content of comments in the future.

Yes, there is! Ideally Eldiro would eventually get tooling built for it, e.g. a language server. Any feature that has to interact with the source file directly (e.g. automatic refactorings or ‘expand selection’) needs to have information about every single character in the source. To accomplish this, the parser has to be lossless, meaning that its output represents the input fully. Although omitting whitespace, comments and other trivia from the parser output would make it more concise and easier to work with, it makes implementing those features that need to interact with the source text much more difficult.
Someone else has also asked this question on Reddit, so I think I’ll add an explanation to the post.

I’ve added an explanation to the website.

Can I close this issue?

Sorry for the late answer, you can close this issue, question answered :)
Thank you very much!

Glad I could help :)


Documentation: clarify the behavior of the label function

Currently, the label docs states:
"The preceding element must be in the same scope as the label, which means that Hello #[<label>], for instance, wouldn't work."
The given example, which includes a use of the label function to create a label, is:
#show <a>: set text(blue)
#show label("b"): set text(red)

= Heading <a>
*Strong* #label("b")

And the docs also states "This function also has dedicated syntax", implying that the non-dedicated syntax should work. All these elements led me to believe that the following code would work:
#set heading(numbering: "1.")
= Heading #label("mylabel")
@mylabel

...but compiling it with typst 0.10, I get the following error:
error: cannot reference text
  ┌─ file.typ:5:0
  │
5 │ @mylabel
  │ ^^^^^^^^

I'm guessing that this use of the label function counts as the content not being "in the same scope as the label"? But if so, does it mean that the label function itself simply cannot be used to create labels for later referencing through the ref or @ syntax? If this is the case, it probably shouldn't be used in the documentation code example I quoted above. I find the documentation a bit confusing on this.

The difference here is syntactical. The <a> terminates the heading and thus attaches to the heading. The #label(..) is part of the heading and thus attaches to the text. Hence, the error message cannot reference text.


"dawkins"

Oh my goodness how did I just leave that typo sitting there for three months. Thanks for catching that! :)


ci: try gcloud resultstore

Signed-off-by: Lizan Zhou<EMAIL_ADDRESS>

@alyssawilk had to revert #21073 because the invocation ID and URLs are printed in as INFO.

actually cc @keith is there trivial a way to have the info logs silenced for the build phase and not for the test phase?


prom: Add option to specify metrics prefix

It's useful to be able to prefix the metrics that are being reported so as to more easily differentiate them from other applications that are using the same metrics.
This change adds the necessary bits (environment var, config file, command line) to support capturing the prefix and passing it through to the httpmetrics library.

@mostynb can you take a look at this?

How disruptive would it be to just hard-code a "bazel_remote" prefix, and skip the configuration setting? And would that be sufficient, if we ignore the distruption for a moment?

For folks that have dashboards and such depending on this it'd probably be fairly disruptive.  If the prefix is empty it shows up as it does today which reduces disruption.

OK, this sounds reasonable. I just have a couple of questions about the flag...
Is there a recommended value? Would "bazel_remote_" work for anyone who might want to use this setting? If so we could use a boolean flag and if there's ever a bazel-remove v3 we could switch to this behaviour and remove the flag.

That is a possible option - I was mostly trying to give maximum flexibility.  Any particular reason for preferring that?

I would like to limit the number of unnecessary configuration settings, because the number of settings is growing more than I would like. If we were to remove some configuration settings in a major version upgrade in the future, then the fewer broken configurations the better, since the migration path for users would be simpler. So if there is no plausible need to specify a custom prefix there, let's not add one.
On the other hand, if you can think of a reason why we would need this configurability then that would change things a bit.

Naw.  That seems totally reasonable.  My initial instinct was that may folks want to prefix this per instance, but after some thought I concluded that probably per application is fine.

Thanks- I squashed this and landed it on the master branch.


setTransform(matrix) missing for CanvasRenderingContext2D and CanvasPattern

for CanvasRenderingContext2D the matrix variant is missing
https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D/setTransform
for CanvasPattern the method is missing
https://developer.mozilla.org/en-US/docs/Web/API/CanvasPattern/setTransform

Source: https://github.com/google/closure-compiler/blob/master/externs/browser/html5.js
upstream of https://github.com/google/elemental2/issues/145

CanvasRenderingContext2D extends BaseRenderingContext2D
https://github.com/google/closure-compiler/blob/5f0ce35c6ccf72ffcda8bf28b8ab155b7de04b59/externs/browser/html5.js#L838-L843
From which it inherits setTransform
https://github.com/google/closure-compiler/blob/5f0ce35c6ccf72ffcda8bf28b8ab155b7de04b59/externs/browser/html5.js#L443-L444
However, we don't seem to have any prototype properties defined for CanvasPattern at all.
https://github.com/google/closure-compiler/blob/5f0ce35c6ccf72ffcda8bf28b8ab155b7de04b59/externs/browser/html5.js#L873
@zbynek would you be interested in creating a pull request to add one?

@brad4d thanks for the reply! The problem with BaseRenderingContext2D.prototype.setTransform
is that it expects 6 number parameters, even though the standard also allows just one parameter of the DOMMatrixInit type.
I'm not sure what's the best way to describe both the old and the new syntax in the extern: maybe the first argument needs to be number|DOMMatrixInit and all the others marked as optional, but that seems to be sub-optimal for tools that process the  externs. @jDramaix would that generate 12 different syntaxes for Elemental2 (two for each number of arguments)?
If the 5 optional arguments are the best we can do, I can create a PR for that.

Looks like we need to also update the documentation links in html5.js
It currently points at
https://www.w3.org/TR/2dcontext/#canvasrenderingcontext2d
But the current standard (found via MDN) seems to be at
https://html.spec.whatwg.org/multipage/canvas.html

Yes, I think 5 optional parameters is really the only way to declare the new single-argument version of setTransform in the externs.


Interactive colorbars

Feature request: enable left-dragging on numbers in colorbar to adjust contrast and overlay thresholds?

Active development moved to MRIcroGL1.2


Add adjustable transaction fee

The transaction fee (a ratio of sats to vbytes) is currently hard coded at 1 sat/vbyte. We should offer the opportunity to users to change that fee, particularly since understanding transaction fees is part of the tutorials.

Duplicate #36.


Cairo1 ci

Disable behaviour tests, cli tests and semantic tests as of now , since cairo1 contracts can't be deployed atm.
Ignore changes  of *.sol files, those changes are just copy/paste of existing solidity contracts

Please add some documentation to the readme


Update Supervision instructions and specifications

Hi, I'm trying to implement Eternal but I get deprecation errors because of this line
      supervisor(Eternal, [:cart_item_lookup, [ :compressed ], [ quiet: true ]]),
asi in here

this is the error

Supervisor.Spec.supervisor/2 is deprecated. Use the new child specifications outlined in the Supervisor module insteadElixir

if I remove supervisor and start it as pointed in the docs of supervisor like this
 {Eternal, [:cart_item_lookup, [ :compressed ], [ quiet: true ]]},
it has another error that it is not a child process or something like that.
This has been changed for a long time apparently since Elixir 1.5 and we're on 1.7, can this somehow be re-descirbed on the README or docs as to how to implement it without errors?

Hi @maikelthedev!
You're right; this library has been largely unchanged going back to v1.5 (it's small+feature complete), so the instructions were outdated. I've updated this in the README, but I'm going to leave this issue open because there are some other improvements I can make in the library itself for the new Supervision setup.
Thank you for filing this issue 🎉

thank you @whitfin


Example toolbar for embed image, video

Hi team,
Do you have source code example for image, video in toolbar?
Please send it to me.
Thanks you!

Support for youtube, vimeo etc links is now fully supported in version 1.2.1.
You can find examples on these links
https://github.com/romainleduc/draft-js-wysiwyg/tree/master/examples/embedded-link)
https://draft-js-wysiwyg.com/components/button
Hope this helps you :)


Problem with code on page

Challenge Learn how Script Tags and Document Ready Work has an issue.
User Agent is: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36.
Please describe how to reproduce this issue, and include links to screenshots if possible.
My code:

jQuery Playground

#left-well

        <button class="btn btn-default" id="target1">#target1</button>
        <button class="btn btn-default" id="target2">#target2</button>
        <button class="btn btn-default" id="target3">#target3</button>

#right-well

        <button class="btn btn-default" id="target4">#target4</button>
        <button class="btn btn-default" id="target5">#target5</button>
        <button class="btn btn-default" id="target6">#target6</button>

The onscreen browser says "Chrome detected unusual code on this page and blocked it to protect your personal information (for example, passwords, phone numbers, and credit cards)."
What's that all about?
Thanks,
Larry

@lclyman closing as duplicate of https://github.com/freeCodeCamp/freeCodeCamp/issues/13727.


Catch globus 404s and more informative logging

if a globus task ID returns 404, mark it as not found in db and stop rechecking.
if influxDB system is down, skip remaining tasks for this iteration.

the issue seems resolved. active task count back to zero and the system is communicating with globus again.


Cronjob error because column end is NULL

Hello,
Actual Behavior
I noticed that the cron jobs change the status to inactive if the database column end is NULL. The database column end should not have the value NULL, because in some cases the cron job aborts. If the option "Disable on errors" is activated but no e-mail address is stored, the administrator will not be informed.
Expected Behavior
When executing the conjob, you should first check whether the column end has the value NULL or not.
Steps to Reproduce
Unknown
Possible Solutions
Whether the cron job is executed correctly or it is aborted due to an error, the cron job in the database should always be saved correctly.
A semantic example:
try {
#run cron
} catch {
#catch errors
} finally {
#save cronjob in database
}

@michael-rusch the installer sets the end column to the date and time of the installation. After that its up to shopware how to handle the actual cronjob. This looks like a shopware bug to me. Which Version of shopware are you using?

Shopware 5.4.6, Connector 4.6.0


Duplicate API auth provider after api update

Describe the bug
Changing an additional auth type to be the default auth type, causes it to be set as both default and additional, if selecting 'No" to 'Configure additional auth types?' when running api update.
Amplify CLI Version
4.28.2
To Reproduce
amplify add api
? Please select from one of the below mentioned services: **GraphQL**
? Provide API name: **amplifybugtest**
? Choose the default authorization type for the API **API key**
? Enter a description for the API key:
? After how many days from now the API key should expire (1-365): **7**
? Do you want to configure advanced settings for the GraphQL API **Yes, I want to make some additional changes.**
? Configure additional auth types? **Yes**
? Choose the additional authorization types you want to configure for the API **IAM**
? Configure conflict detection? **No**
? Do you have an annotated GraphQL schema? **No**
? Choose a schema template: **Single object with fields (e.g., “Todo” with ID, name, description)**

amplify update api
? Please select from one of the below mentioned services: **GraphQL**
? Select from the options below **Update auth settings**
? Choose the default authorization type for the API **IAM**
? Configure additional auth types? **No**

The authConfig in amplify-meta.json now look like this:
"authConfig": {
  "defaultAuthentication": {
    "authenticationType": "AWS_IAM"
  },
  "additionalAuthenticationProviders": [
    {
      "authenticationType": "AWS_IAM"
    }
  ]
}

And 'amplify push' results in:
The following resource(s) failed to create: [GraphQLAPI]
Authentication type AWS_IAM for additional authentication provider 1 already specified on the API. It can only be specified once. (Service: AWSAppSync; Status Code: 400; Error Code: BadRequestException;
Expected behavior
AWS_IAM authentication type to be removed from additionalAuthenticationProviders
Desktop (please complete the following information):

OS: Mac
Node Version: v14.9.0

Hi @11csive
Thanks for reporting this. I have reproduced it.


(R)MP2 gradient inefficiency due to einsum implementation

Hi devs!
When I try to calculate benzene (C6H6) gradient at MP2/cc-pVTZ level, I found default pyscf (obtained from pypi, where numpy/mkl from conda) often run in only one thread (100% CPU) where multiple threads available (e.g. 24 CPU).
This problem is probably from einsum function implementation. Algorithms are correct.
The possible solution will be stated in next reply.
The code to perform a gradient could be:
from pyscf import gto, scf, mp, grad

mol = gto.Mole()
mol.atom = """
C   1.40000000   0.00000000   0.00000000
C   0.70000000   1.21243557   0.00000000
C  -0.70000000   1.21243557   0.00000000
C  -1.40000000   0.00000000   0.00000000
C  -0.70000000  -1.21243557   0.00000000
C   0.70000000  -1.21243557   0.00000000
H   2.49000000   0.00000000   0.00000000
H   1.24500000   2.15640326   0.00000000
H  -1.24500000   2.15640326   0.00000000
H  -2.49000000   0.00000000   0.00000000
H  -1.24500000  -2.15640326   0.00000000
H   1.24500000  -2.15640326   0.00000000
"""
mol.basis = "cc-pVTZ"
mol.max_memory = 20 * 1024
mol.build()

mf_scf = scf.RHF(mol).run()
mf_mp2 = mp.MP2(mol).run()
mf_grad = grad.mp2.Gradients(mf_mp2).run()

You can try to compile the TBLIS interface when compiling the C extensions.  http://pyscf.org/pyscf/install.html#tblis
pyscf/pyscf/mp/mp2.py L191 can be replaced by lib.einsum. It can be faster than numpy einsum
L99 to L110 are memory bounded. lib.einsum is likely not helpful.
L96 in principle can be improved if the tblis plugins are compiled.

Thanks!
For grad/mp2.py L96, actually I've tried out TBLIS. That is quite efficient 😄 (the compromise is to use pyscf from source but not pypi)
I found that numpy could be equally efficient if atomic basis is large, but the compromise is an explicit for loop, thus complexity of code is increased.
I just wonder in case of this situation, if you would perfer to change the code, or keep the code as it was 😂

It'd be great if you can issue a PR for the optimization.
In the upcoming pyscf-2.0 release, the pyscf-TBLIS interface will be released as
a precompiled wheel. The tblis-optimized einsum will be available in pip. The
TBLIS functionalities can be turned on/off by setting in the code
pyscf.lib.numpy_helper.FOUND_TBLIS = True  # False
On Wed, Mar 10, 2021 at 06:36:04PM -0800, Zhenyu Zhu ajz34 wrote:

Thanks!
For grad/mp2.py L96, actually I've tried out TBLIS. That is quite efficient 😄 (the compromise is to use pyscf from source but not pypi)
I found that numpy could be equally efficient if atomic basis is large, but the compromise is an explicit for loop, thus complexity of code is increased.
I just wonder in case of this situation, if you would perfer to change the code, or keep the code as it was 😂
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/pyscf/pyscf/issues/835#issuecomment-796388194

--

Wow, that sounds great! 🎉


KOGITO-7386: Implement acceptance tests for collection with generics in BPMN editor

Adds test and helper classes.
Tested locally as this test suite is not enabled.
JIRA: https://issues.redhat.com/browse/KOGITO-7386
Feature JIRA: https://issues.redhat.com/browse/BAPL-2027

Working on the rebase will let you know.

Rebased @tiagobento


Can't compress bitmap stored as byte[] using lz4net (Version:<IP_ADDRESS>)

I have been trying to use the available NuGet package of lz4net (Version:<IP_ADDRESS>) to compress a bitmap quickly. Motivation is to log incoming bitmaps to disk from a Frame grabber which dispenses frames every 30 ms.
Here is the concerning code snippet
    private void Lz4Compression()
    {
        try
        {
            using (FileStream fs = new FileStream(m_outputPath, FileMode.Create))
            {
                using (LZ4Stream stream = new LZ4Stream(fs, LZ4StreamMode.Compress))
                {
                    using (BinaryWriter writer = new BinaryWriter(stream))
                    {
                        writer.Write(m_loadedImage);
                    }
                }
            }
        }
        catch (Exception e)
        {
            Console.WriteLine(e);
        }
    }

The resulting compressed file is of "almost" the same size in bytes as the original bitmap loaded as a byte[], m_loadedImage.
Does the compression not work when using a BinaryWriter instead of a StreamWritter?
When I followed the Lorem Ipsum example, text only, I had great compression results.
Any feedback would be greatly appreciated.

As mentioned on front page LZ4 is fast but "sacrificing compression ratio for compression/decompression speed".

Lorem Ipsum example compresses well as it is same text repeated many times
Use high compression (it is slower, but gives better compression ratio)
You can try increasing block size

On top of that:

Don't use BinaryWriter just use Stream directly (does not affect size though, it is just a performance)

Very much appreciate the feedback! I've made your suggested changes, but as you have highlighted I am still encountering the compression ratio issue using the following code.
 private void Lz4Compression()
    {
        try
        {
            using (FileStream fs = new FileStream(m_outputPath, FileMode.Create))
            {
                using (LZ4Stream stream = new LZ4Stream(fs, LZ4StreamMode.Compress, Z4StreamFlags.HighCompression, 1048576 * 5))
                {
                    stream.Write(m_loadedImage, 0, m_loadedImage.Length);
                }
            }
        }
        catch (Exception e)
        {
            Console.WriteLine(e);
        }
    }

I will investigate alternative options. Thanks!


Modularize pixi-react to allow user defined versions of React and PIXI

Spec
Rough map of current source to new modules

components -> pixi-react-components
helpers -> pixi-react
hoc + stage/provider-> pixi-react?
hooks -> pixi-react-components
reconciler -> pixi-react-fiber
render -> pixi-react-components
stage/index -> pixi-react-components
utils/element -> pixi-react-fiber
utils/element/PixiComponent -> pixi-react
utils/invariant -> pixi-react
utils/pixi -> pixi-react-components?
utils/props -> pixi-react-components?

Notes

We can probably rely on peer-deps to handle React imports, but anything that relies on a PIXI import most likely needs to exist within pixi-react-components to maximize compatibility (since the PIXI import API has changed considerably between versions).
At first glance it would seem that pixi-react-components should be thin and pixi-react should be fat, but it likely needs to be the other way around. pixi-react will pretty much expose only the PixiComponent custom component API and possibly some utils.
Virtually everything else will come from pixi-react-components bar the React Reconciler, which comes from pixi-react-fiber.
The React Reconciler also needs configuration, although most is static, createElement depends on TYPES_INJECTED which will be stored in pixi-react but filled up in pixi-react-components.

Proposed Usage
In a user's app they should setup pixi-react something like this:
import { configurePixiReact } from 'pixi-react';
import { configurePixiReactFiber } from 'pixi-react-fiber-v18';
import { configurePixiReactComponents } from 'pixi-react-components-v7';

export default configurePixiReact({
    configurePixiReactComponents,
    configurePixiReactFiber,
});

Implementation
The configuration function would look something like this:
function configurePixiReact({
    configurePixiReactComponents,
    configurePixiReactFiber,
})
{
    // PixiComponent updates TYPES_INJECTED (or new name since all will be), but this can basically be private and not
    // allow dupes?
    // invariant, PixiComponent, CHILDREN and TYPES_INJECTED come from pixi-react

    // getTextureFromProps also lives inside pixi-react-components even if not exported
    const {
        TYPES,
        createRoot,
        render,
        unmountComponentAtNode,
        Stage,
        applyDefaultProps,
        AppProvider,
        AppConsumer,
        AppContext,
        useTick,
        useApp,
        withPixiApp,
        withFilters,
        eventHandlers,
    } = configurePixiReactComponents({
        invariant,
        PixiComponent,
    });

    // All of the reconciler is basically static, except createElement which depends on TYPES_INJECTED, this configure step
    // basically just makes createElement and then wires everything up
    const PixiFiber = configurePixiReactFiber({
        CHILDREN,
        // updated to include pixi-react-components by configurePixiReactComponents, should we pass this around explicitly?
        TYPES_INJECTED,
        invariant,
        applyDefaultProps,
    });

    return {
        // From pixi-react
        PixiComponent,
        // From pixi-react-components
        TYPES,
        createRoot,
        render,
        unmountComponentAtNode,
        Stage,
        AppProvider,
        AppConsumer,
        AppContext,
        useTick,
        useApp,
        applyDefaultProps,
        // These could come from pixi-react, should they tie to PIXI or React version?
        withPixiApp,
        withFilters,
        eventHandlers,
        // From pixi-react-fiber
        PixiFiber,
    };
}

@Zyie once we release this I think perhaps we should write a note about what we officially support? We can obviously encourage users to write whatever pixi component packages they need for whichever versions of PIXI, but the combinations could get complicated quickly.
So far we've talked about supporting React 17+ and PIXI 6+, does that mean we support:

React 17 + PIXI 6
React 18 + PIXI 6
React 17 + PIXI 7
React 18 + PIXI 7

Combinations could obviously get complex quickly especially as we versions continue to bump.
I'm wondering whether on top of the user injectable API I've proposed above, we should also provide pre-configured pixi-react version(s) packaged as a single module. So for example @pixi/react v8.0.0 becomes a wrapper for the above API including React 18 and PIXI 7?

@Zyie once we release this I think perhaps we should write a note about what we officially support? We can obviously encourage users to write whatever pixi component packages they need for whichever versions of PIXI, but the combinations could get complicated quickly.
So far we've talked about supporting React 17+ and PIXI 6+, does that mean we support:

React 17 + PIXI 6
React 18 + PIXI 6
React 17 + PIXI 7
React 18 + PIXI 7

Yeah i think these should be what we support

So for example @pixi/react v8.0.0 becomes a wrapper for the above API including React 18 and PIXI 7?

Yeah this is a good approach. Makes it very simple for whoever wants the latest version


SB-791: Another Windows path issues fixed

There is only one failed test for me under Windows.
@fuss86  can you check it with your environment?
its ArtifactDirectoryLocatorTest and the error below:
-------------------------------------------------------------------------------
Test set: org.carlspring.strongbox.artifact.locator.ArtifactDirectoryLocatorTest
-------------------------------------------------------------------------------
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.766 sec <<< FAILURE!
testLocateDirectories(org.carlspring.strongbox.artifact.locator.ArtifactDirectoryLocatorTest)  Time elapsed: 0.188 sec  <<< ERROR!
java.io.UncheckedIOException: java.nio.file.AccessDeniedException: Z:\strongbox\strongbox\strongbox-storage\strongbox-storage-layout-providers\strongbox-storage-maven-layout-provider\target\strongbox-vault\storages\storage0\mcs-releases\org\carlspring\strongbox\checksum\maven\strongbox-checksum\maven-metadata.xml
	at java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:88)
	at java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:104)
	at java.util.Iterator.forEachRemaining(Iterator.java:115)
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
	at org.carlspring.strongbox.artifact.locator.handlers.AbstractArtifactLocationHandler.getVersionDirectories(AbstractArtifactLocationHandler.java:44)
	at org.carlspring.strongbox.artifact.locator.handlers.ArtifactLocationReportOperation.execute(ArtifactLocationReportOperation.java:72)
	at org.carlspring.strongbox.artifact.locator.ArtifactDirectoryLocator.execute(ArtifactDirectoryLocator.java:85)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.util.ArrayList.forEach(ArrayList.java:1249)
	at java.util.stream.SortedOps$RefSortingSink.end(SortedOps.java:390)
	at java.util.stream.Sink$ChainedReference.end(Sink.java:258)
	at java.util.stream.Sink$ChainedReference.end(Sink.java:258)
	at java.util.stream.Sink$ChainedReference.end(Sink.java:258)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
	at org.carlspring.strongbox.artifact.locator.ArtifactDirectoryLocator.locateArtifactDirectories(ArtifactDirectoryLocator.java:39)
	at org.carlspring.strongbox.artifact.locator.ArtifactDirectoryLocatorTest.testLocateDirectories(ArtifactDirectoryLocatorTest.java:211)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
	at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:252)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:94)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
	at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:191)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: java.nio.file.AccessDeniedException: Z:\strongbox\strongbox\strongbox-storage\strongbox-storage-layout-providers\strongbox-storage-maven-layout-provider\target\strongbox-vault\storages\storage0\mcs-releases\org\carlspring\strongbox\checksum\maven\strongbox-checksum\maven-metadata.xml
	at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
	at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
	at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
	at sun.nio.fs.WindowsFileAttributeViews$Basic.readAttributes(WindowsFileAttributeViews.java:53)
	at sun.nio.fs.WindowsFileAttributeViews$Basic.readAttributes(WindowsFileAttributeViews.java:38)
	at sun.nio.fs.WindowsFileSystemProvider.readAttributes(WindowsFileSystemProvider.java:193)
	at org.carlspring.strongbox.providers.io.RepositoryFileSystemProvider.readAttributes(RepositoryFileSystemProvider.java:458)
	at java.nio.file.Files.readAttributes(Files.java:1737)
	at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
	at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
	at java.nio.file.FileTreeWalker.next(FileTreeWalker.java:372)
	at java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:95)
	... 63 more

@carlspring your opinion need here too

@sbespalov here is my report from your branch:
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Strongbox: Parent .................................. SUCCESS [  0.692 s]
[INFO] Strongbox: Resources [Common] ...................... SUCCESS [  1.317 s]
[INFO] Strongbox: Resources [Storage API] ................. SUCCESS [  0.186 s]
[INFO] Strongbox: Resources [Storage] ..................... SUCCESS [  0.034 s]
[INFO] Strongbox: Resources [Web] ......................... SUCCESS [  0.437 s]
[INFO] Strongbox: Resources ............................... SUCCESS [  0.020 s]
[INFO] Strongbox: Data Service ............................ SUCCESS [  2.954 s]
[INFO] Strongbox: Commons ................................. SUCCESS [  4.736 s]
[INFO] Strongbox: Configuration ........................... SUCCESS [  0.690 s]
[INFO] Strongbox: Metadata [Maven API] .................... SUCCESS [  2.082 s]
[INFO] Strongbox: Event API ............................... SUCCESS [  0.289 s]
[INFO] Strongbox: Authentication API ...................... SUCCESS [  0.327 s]
[INFO] Strongbox: Client .................................. SUCCESS [  5.485 s]
[INFO] Strongbox: Testing [Core] .......................... SUCCESS [  1.191 s]
[INFO] Strongbox: Security API ............................ SUCCESS [ 14.897 s]
[INFO] Strongbox: Storage [Core] .......................... SUCCESS [  2.108 s]
[INFO] Strongbox: Testing [Storage] ....................... SUCCESS [  1.177 s]
[INFO] Strongbox: Storage [API] ........................... SUCCESS [ 32.926 s]
[INFO] Strongbox: User Management ......................... SUCCESS [ 19.180 s]
[INFO] Strongbox: REST Client ............................. SUCCESS [  1.215 s]
[INFO] Strongbox: Storage [Maven Layout Provider] ......... FAILURE [01:13 min]
[INFO] Strongbox: Testing [Web] ........................... SKIPPED
[INFO] Strongbox: Testing ................................. SUCCESS [  0.016 s]
[INFO] Strongbox: Storage [Nuget Layout Provider] ......... SUCCESS [  1.809 s]
[INFO] Strongbox: Storage [P2 Layout Provider] ............ SUCCESS [  2.172 s]
[INFO] Strongbox: Storage Layout Providers ................ SUCCESS [  0.015 s]
[INFO] Strongbox: Storage ................................. SUCCESS [  0.016 s]
[INFO] Strongbox: Web Core ................................ SKIPPED
[INFO] Strongbox: Cron Tasks .............................. SKIPPED
[INFO] Strongbox: Masterbuild ............................. SUCCESS [  0.016 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:50 min
[INFO] Finished at: 2017-05-28T23:30:09+02:00
[INFO] Final Memory: 81M/582M
[INFO] ------------------------------------------------------------------------

and the error is:
testRewriteMavenChecksum(org.carlspring.strongbox.services.MavenChecksumServiceTest)  Time elapsed: 0.089 sec  <<< ERROR!
java.io.UncheckedIOException: java.nio.file.AccessDeniedException: C:\Users\fuss\IdeaProjects\strongbox\strongbox-storage\strongbox-storage-layout-providers\strongbox-storage-maven-layout-provider\target\strongbox-vault\storages\storage0\mcs-releases\org\carlspring\strongbox\checksum\maven\strongbox-checksum\maven-metadata.xml
        at java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:88)
        at java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:104)
        at java.util.Iterator.forEachRemaining(Iterator.java:115)
        at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
        at org.carlspring.strongbox.artifact.locator.handlers.AbstractArtifactLocationHandler.getVersionDirectories(AbstractArtifactLocationHandler.java:44)
        at org.carlspring.strongbox.artifact.locator.handlers.ArtifactLocationGenerateChecksumOperation.execute(ArtifactLocationGenerateChecksumOperation.java:81)
        at org.carlspring.strongbox.artifact.locator.ArtifactDirectoryLocator.execute(ArtifactDirectoryLocator.java:85)

I've built running command:  mvn clean install -fae

Not all the tests can be omni-potent so some of them have the test data for the various test methods all set up via a @PostConstruct method. I have refactored these tests a lot to even get them into this state, as they used to be a lot more entagled before. I don't think these methods that set up the resources can be reworked much further.

@carlspring , @sbespalov all issues resolved in https://github.com/sbespalov/strongbox/pull/1
Today I found couple of resource leaks in the source code. All are fixed.

@carlspring I think it can be merged now.
@fuss86 thanks for your help.

@carlspring @fuss86 One more thing.
I think that as a part of SB-757 we should remove all the initialized flags from the tests, to be sure that we have no resource leeks any more.

Hi,
Yeah, I agree. If things are really fixed, this should worm without the
need for those initialization checks.
On 31 May 2017 04:03, "Sergey Bespalov"<EMAIL_ADDRESS>wrote:

@carlspring https://github.com/carlspring @fuss86
https://github.com/fuss86 One more thing.
I think that as a part of SB-757 we should remove all the initialized
flags from the tests, to be sure that we have no resource leeks any more.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/strongbox/strongbox/pull/273#issuecomment-305069083,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABXqaZ4GvpKHj-U5DaLXnPEZf9SMWLnpks5r_NiXgaJpZM4NoQgJ
.

I think that as a part of SB-757 we should remove all the initialized flags from the tests, to be sure that we have no resource leaks any more.

If things are really fixed, this should work without the need for those initialization checks.

Guys, first initialized flag added to test, was related to file locking problem:

org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@D:\workspace_neon\workspace_strongbox\strongbox\strongbox-storage\strongbox-storage-layout-providers\strongbox-storage-maven-layout-provider\target\strongbox-vault\storages\storage0\m2lp-releases.index\local\write.lock: java.nio.channels.OverlappingFileLockException

It's possible that it was caused by resource leak in strongbox code (like unclosed stream), some inproper resources handling in strongbox code (like double initialization) or internals of apache lucene. initialized  flag was a quick "solution" for this problem.
Then it turns out that it's a "solution" for more test errors under Windows. It means that every initialized  flag introduced by me may be related with completely different cause.
Maybe I was wrong but I assumed that the expected outcome is to find a way to solve these problems ASAP. Of course the root problems lie somewhere underneath, it's obvious. I'm still getting familiar with this project and 1,5 months ago was my first code overview.
Please also have in mind that first initialized flag in the *Test.java code wasn't mine. How was I supposed to know what's legal in the project and how much you want to know about the root cause of the problem ?

@carlspring I think it can be merged now.

yes, it can be merged now

@carlspring @fuss86 One more thing.
I think that as a part of SB-757 we should remove all the initialized flags from the tests, to be sure that we have no resource leaks any more.

we can open new branch for that and re-open SB-757

@fuss86,
First of all, nobody's judging your work here. Your work is great!
We know that some of the test cases are far from optimal. Your first task was to sort out the Windows build issues and after that to apply the necessary changes to our code so that the latest maven-indexer:6.0-SNAPSHOT would work for our code base. You did both tasks. However, with the upgrade to the latest maven-indexer, these lock issues started popping up. I would argue that sorting them out properly would be part of the original task relating to the upgrade.
It seems that there might possibly be duplicate index initializations, or something else that locks the resources.
I know that this turned out to be a lengthy task. If you think this should be handled in a separate follow-up task, then please message me on Skype so we can work out the details.
Many thanks for your efforts!

However, with the upgrade to the latest maven-indexer, these lock issues started popping up.

It's not true. maven-indexer upgrade, once merged, did not bring any new issues to the project. Please review my comments at SB-757. I've started working on Windows issues about 11 Apr 2017. With the "old"  maven-indexer on board, there was one bigger problem that required deep analysis of maven indexer, see my comment from 19 Apr 2017:

Unfortunately I'm unable to prepare a quick fix for above error as it requires very deep analysis inside the maven indexer. Because the decision has been made, that the maven indexer will be upgraded soon - it makes no sense to waste the time for that any longer and handle this internal problem, as there is a possibility it'll be resolved in the lateset maven indexer release.

I've upgraded maven-indexer and strongbox project has been left on the same state, before and after maven-indexer upgrade. The problem was there before and maven-indexer upgrade did not fix it.

Okay. In this case, I propose we have a follow-up task that looks into the proper resolution of this issue.
Would you like to create it (with a proper description and issue links) and you can pick it up when we have time for it?


[TB] Week of March 22

Draft email to Blei and send to Amine and Hassan:

Deliverable: Email to Amine and Hassan
Estimate:
Actual:

Email Amine the code for simulating from model in pylogit:

Deliverable: pushed code to reference folder in separate branch
Estimate:
Actual:

[ ] Weekly Meeting:

Estimate: 2 hours
Actual:

@timothyb0912 feel free to add estimates in there.

@bouzaghrane for code showing simulation from a pylogit model, see cell 7 here: https://github.com/adb40/machine_learning_session/blob/TimB-files/src/scripts/_03_tb-MNL_Checking.ipynb
You can grab the needed functions from here: https://github.com/adb40/machine_learning_session/tree/TimB-files/src/scripts/viz
and in particular from the function here: https://github.com/adb40/machine_learning_session/blob/TimB-files/src/scripts/viz/utils.py#L225


Preventing script cashing on updates

Now we have all this great functionality to pull down client side packages, like bootstrap.js, etc..
Problem arises on updates, and the issue is script caching.
One wants the script to be cached, until on update.
One way to deal with this is put the version number as a parameter
src="~/lib/bootstrap/js/bootstrap.js?v=3.3.2
So I am wondering could there be a class library which will automatically get updated based on the package downloaded, so I can do a call
src="~/lib/bootstrap/js/bootstrap.js?v=@(Packages.BoostrapJS.Version)
Another issue is the site itself, version-ing of the local scripts. If my site is packaged as MySite1.0.0. In MVC3 this was achieved aka
@typeof(YourApplicationNamespace.MvcApplication).Assembly.GetName().Version
How can I get the version in  VNEXT

This issue is being closed because it has not been updated in 3 months.
We apologize if this causes any inconvenience. We ask that if you are still encountering this issue, please log a new issue with updated information and we will investigate.


Add Axl+ dendritic cell

Related to issue #764.
I was not able to think about a logical definition, maybe @addiehl or @masciam have suggestions?

FYI @shawntanzk

2 things need to be added:

[ ] in_taxon some homo_sapien
[ ] expresses AXL, expresses SIGLEC6 <- CL uses PRO atm despite inaccuracy of protein vs gene (might be related to this PR https://github.com/obophenotype/cell-ontology/pull/901)

@shawntanzk in_taxon some homo_sapien is inferred from subclassOf "dendritic cell, human", isn't it?

@lubianat - sorry, my bad, youre right :)

@lubianat, checking in on this PR. It has no reviewers and seems to have no activity for several months. Can you merge or close?

@bvarner-ebi thanks! @shawntanzk pointed I should add some way of saying the cell expresses the genes. How should I do that, though?
I think both proteins are present in the plasma membrane at some point, but is there a more general way that does not depend on the protein in the plasma membrane?

@bvarner-ebi thanks! @shawntanzk pointed I should add some way of saying the cell expresses the genes. How should I do that, though?

expresses some PR:000001928
expresses some PR:000004528

Let me know if you need help adding those in the PR obo slim :)

@shawntanzk  I don't know what you mean by "OBO slim", I'm sorry

By PR you mean Protein Ontology or Pull Request? lol

@shawntanzk I don't know what you mean by "OBO slim", I'm sorry

ah yeah, we really need to document this.
Basically PR ontology is too big, so we have a obo slim version where we add terms we care about: https://github.com/obophenotype/pro_obo_slim
We import that instead of the whole PR, so new terms might have to be added there before we import them into CL.
if this isn't urgent, give me some time to write docs
carpal tunnel is killing me a bit atm, so writing full docs sounds painful >.<

@shawntanzk Not urgent for me; take care of your carpal tunnel! It looks a bit cumbersome, so I'll wait for the docs. Thanks!

@lubianat, it's been a while since I did this, but here are my notes on adding PRO terms to CL after @shawntanzk helped me do this.
If you try this, can you let us know how it goes? If successful, they can be added to documentation...

Add desired PR term to seed.txt in pro_obo_slim
Open docker
Open terminal
Navigate (cd) to the main directory in pro_obo_slim
Run sh odk.sh make all
Make a pull request and merge all updates - can merge this without review
In the cell-ontology repo, create a branch
Add PRO term to src/ontology/imports/pr_terms.txt
Go to terminal, navigate to cell-ontology/src/ontology
Run sh run.sh make mirror/pr.owl -b
(Note: -b forces command to run even if the mirror is up-to-date)
Run sh run.sh make imports/merged_import.owl
Open Protégé. The PRO term should now be available for modelling
END

@Thank you bvarner!
We should have a simpler way to automate all of that...
My step-by-step:
After having Docker installed and configured:

Clone https://github.com/obophenotype/pro_obo_slim.
Edit locally https://github.com/obophenotype/pro_obo_slim/blob/master/seed.txt to add the Protein Ontology terms desired.
Navigate (cd) to the main directory in pro_obo_slim
Run sh odk.sh make all to download docker image and run code to update the Slim version of PR

At this point I had to stop, because the diff has added ^^xsd:string statements. This is a known issue:

https://obophenotype.github.io/cell-ontology/Fixing_xsdstring_diffs/

But I am not sure on how to do it for pro_obo_slim

Hi @lubianat, do you use macos? You can try
sed -i '' -E "s/Annotation[(](oboInOwl[:]hasDbXref [\"][^\"]*[\"])[)]/Annotation(\1^^xsd:string)/g" pr_slim.owl

@lubianat, if you merge this PR by end of day today, it can be incorporated for the next CL release, which is planned for tomorrow. Otherwise, it would go in a future release.

Hi, @lubianat, can we close this PR?
If there are pending tech issues, can you work with @anitacaron to resolve?

@lubianat, thank you for the update.
May I suggest going forward to use the new term request template and leaving it unassigned?
That way you can still contribute to CL by providing content in the template and an editor can pick up the issue and manage the technical aspects of adding the term. It's an option to continue contributing to CL without navigating the potential technical hurdles.
For the sake of this ticket, I propose:

[ ] close this PR without merging
[ ] @lubianat: double-check #764 and confirm the content is correct
[ ] once #764 is reviewed, assign the ticket to me, and I'll put it in my work queue to add (but may be low priority)

Hi Tiago,

This issue has already taken more than 2 years (!), while on Wikidata I could add it as soon as I desired, taking in a couple minutes on June 2021.

This task, for example, requires a lot of memory usage; I am lucky to have 32 GB RAM in my work computer, 16 wouldn't cut it. It would be good if the hardware requirements were documented; I risked a crash in my PC and losing other stuff I am running.

I agree that this is completely ridiculous. I also think it is unnecessary.  You should be able to do your edits and hand this off to the technical team to do relevant updated before a release.
There is a very simple fix for referring to terms not yet in imports - that is just add a new Class in Protege under root, pasting in the URL (https://oboacademy.github.io/obook/howto/update-import/#protege-based-declaration).    This has been discouraged in recent years because it can cause some pipeline issues, but that should not be something anyone outside of the technical team should need to worry about.
Hope this helps,
David

@anitacaron  to fix the PRO references on this ticket so that it can be merged.

I'll handle it :)

#gogoeditdiff

I'm a bit worried about making a equiv axiom as i don't know how strong the markers are, so I just did subclasses instead, hope that is ok


Setting trainable to false change the order in which the layers of a sub-model are serialized

When using a model within another model and setting some layers with trainable=False change the way the model is serialized.
Then, it is not possible to use load_weights.
Here's an example which results in error:
from keras.models import Model
from keras.layers import Dense, Input, Add

def create_model(frozen=True):
    x = Input(shape=(100,))
    y = Dense(20)(x)
    y = Dense(5, trainable=frozen)(y)
    y = Dense(10)(y)
    model1 = Model(x, y)

    X = Input(shape=(100,))
    Y = model1(X)
    model2 = Model(X, Y)

    return model2

m1 = create_model(True)
m1.save_weights('test.h5')

m2 = create_model(False)
m2.load_weights('test.h5')

which results in this error:
Traceback (most recent call last):
  File "test.py", line 21, in <module>
    m2.load_weights('test.h5')
  File "/home/javier/.local/lib/python3.5/site-packages/keras/engine/topology.py", line 2538, in load_weights
    load_weights_from_hdf5_group(f, self.layers)
  File "/home/javier/.local/lib/python3.5/site-packages/keras/engine/topology.py", line 2970, in load_weights_from_hdf5_group
    K.batch_set_value(weight_value_tuples)
  File "/home/javier/.local/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2148, in batch_set_value
    assign_op = x.assign(assign_placeholder)
  File "/home/javier/.local/lib/python3.5/site-packages/tensorflow/python/ops/variables.py", line 512, in assign
    return state_ops.assign(self._variable, value, use_locking=use_locking)
  File "/home/javier/.local/lib/python3.5/site-packages/tensorflow/python/ops/state_ops.py", line 271, in assign
    validate_shape=validate_shape)
  File "/home/javier/.local/lib/python3.5/site-packages/tensorflow/python/ops/gen_state_ops.py", line 47, in assign
    use_locking=use_locking, name=name)
  File "/home/javier/.local/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op
    op_def=op_def)
  File "/home/javier/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2338, in create_op
    set_shapes_for_outputs(ret)
  File "/home/javier/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1719, in set_shapes_for_outputs
    shapes = shape_func(op)
  File "/home/javier/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1669, in call_with_requiring
    return call_cpp_shape_fn(op, require_shape_fn=True)
  File "/home/javier/.local/lib/python3.5/site-packages/tensorflow/python/framework/common_shapes.py", line 610, in call_cpp_shape_fn
    debug_python_shape_fn, require_shape_fn)
  File "/home/javier/.local/lib/python3.5/site-packages/tensorflow/python/framework/common_shapes.py", line 676, in _call_cpp_shape_fn_impl
    raise ValueError(err.message)
ValueError: Dimension 0 in both shapes must be equal, but are 5 and 20 for 'Assign_2' (op: 'Assign') with input shapes: [5,10], [20,5].

Even when using load_weights(..., by_name=True) the problem persist.

This is a fix when the sub-model name's are the same as the ones saved. The problem persist if no layer names are specified.
diff --git a/keras/engine/topology.py b/keras/engine/topology.py
index 7b27a9b..36019ac 100644
--- a/keras/engine/topology.py
+++ b/keras/engine/topology.py
@@ -3021,6 +3021,7 @@ def load_weights_from_hdf5_group_by_name(f, layers):
         g = f[name]
         weight_names = [n.decode('utf8') for n in g.attrs['weight_names']]
         weight_values = [g[weight_name] for weight_name in weight_names]
+        weight_indices = dict((name, i) for i, name in enumerate(weight_names))

         for layer in index.get(name, []):
             symbolic_weights = layer.weights
@@ -3039,6 +3040,7 @@ def load_weights_from_hdf5_group_by_name(f, layers):
                                  ' element(s).')
             # Set values.
             for i in range(len(weight_values)):
+                weight_index = weight_indices[symbolic_weights[i].name]
                 weight_value_tuples.append((symbolic_weights[i],
-                                            weight_values[i]))
+                                            weight_values[weight_index]))
     K.batch_set_value(weight_value_tuples)

if 2 models' trainable parameter is different, the weights can't be loaded.

Still an issue with keras 2.1.5, copy/paste the code results in the same error


chore(copybara): sync commits from Aspect-internal silo

Commit range https://github.com/aspect-build/silo/compare/f56fd5937e699caf0a8936fed2868de1f1062508..9884bf7f7c1f7611d431b6b1fc2ed049ec64dd03

This was done in another PR


Prefund Verification Script Overflow

The prefund verification script can overflow, and cause the GitHub action to not correctly mark a chain as ready to deploy (source):
Run bash ./.github/scripts/validate_new_chain_request.sh
Extracted RPC url: https://jsonrpc.euphoria.aura.network./ Trying to get the chain id...
6321 from the RPC URL is valid. Checking the chain in chainlist...
Current gas price: 0x7
Estimated deployment gas limit: 0x13f5d
Expected pre-fund: 858445
Deployer address balance: -8446744073709551616
Error: Process completed with exit code 1.

I guess it might be time to rewrite that to JavaScript

I was thinking the exact same thing!

I sent 10^19 gwei and met overflowed minus balance -8446744073709551616 . Yet I sent another 8.5 * 10^18 gwei to let it overflow again, back to positive balance🤣


Fix stdlib compression links

The links for Gzip, Zip, and Zlib were broken. They're now part of Compress.
(I checked the rest of the links to crystal-lang.org and all of the others looked right to me.)

Thank you.


Prototype a new CLI

Demo:
$ cat pyproject.toml
[project]
name = "efbutils"
version = "0"

requires-python = "~=3.8"
dependencies = [
  "amaranth @ file:///${PROJECT_ROOT}/../amaranth"
]

[build-system]
requires = ["pdm-backend"]
build-backend = "pdm.backend"

[tool.pdm.build]
includes = ["efbutils/"]
$ cat Makefile
-include page_buffer.v.d
page_buffer.v:
        pdm run amaranth generate efbutils.ufm.reader.page_buffer:PageBuffer verilog -v $@ -d<EMAIL_ADDRESS>$ make page_buffer.v
pdm run amaranth generate efbutils.ufm.reader.page_buffer:PageBuffer verilog -v page_buffer.v -d page_buffer.v.d
$ make page_buffer.v
make: 'page_buffer.v' is up to date.
$ touch efbutils/__init__.py
$ make page_buffer.v
pdm run amaranth generate efbutils.ufm.reader.page_buffer:PageBuffer verilog -v page_buffer.v -d page_buffer.v.d

I'm quite happy with this PR at this point. But I don't want it merged before other people have tested it. Any chance we could bring it up at the next meeting to let other ppl know to try it out and report issues as a "final comment period"?

Any chance we could bring it up at the next meeting to let other ppl know to try it out and report issues as a "final comment period"?

It's still a draft in my thoughts--I just prototyped it for you. Merging it needs an RFC first.


[SPARK-24646][CORE]  Minor change to spark.yarn.dist.forceDownloadSchemes to support wildcard '*'

What changes were proposed in this pull request?
In the case of getting tokens via customized ServiceCredentialProvider, it is required that ServiceCredentialProvider be available in local spark-submit process classpath. In this case, all the configured remote sources should be forced to download to local.
For the ease of using this configuration, here propose to add wildcard '*' support to spark.yarn.dist.forceDownloadSchemes, also clarify the usage of this configuration.
How was this patch tested?
New UT added.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/455/
Test PASSed.

Test build #92289 has started for PR 21633 at commit f3c99f8.

Test build #92289 has finished for PR 21633 at commit f3c99f8.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92289/
Test FAILed.

Jenkins, retest this please.

Test build #92291 has started for PR 21633 at commit f3c99f8.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/457/
Test PASSed.

Test build #92291 has finished for PR 21633 at commit f3c99f8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92291/
Test PASSed.

Merged build finished. Test PASSed.

Test build #92552 has started for PR 21633 at commit 4419f52.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/630/
Test PASSed.

Test build #92552 has finished for PR 21633 at commit 4419f52.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92552/
Test FAILed.

Jenkins, retest this please.

Test build #92555 has started for PR 21633 at commit 4419f52.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/632/
Test PASSed.

Merged build finished. Test PASSed.

Test build #92555 has finished for PR 21633 at commit 4419f52.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92555/
Test FAILed.

Jenkins, retest this please.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/637/
Test PASSed.

Test build #92563 has started for PR 21633 at commit 4419f52.

Test build #92563 has finished for PR 21633 at commit 4419f52.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92563/
Test PASSed.

Any more comment @vanzin ?

Thanks @jiangxb1987 , merging to master branch.


hacking ideas/improvements

pjb3005 said on reddit it would be good if i left a github issue about this.
/u/HeatLiquid posted this modification to the hacking mock up from the most resent update post. https://i.imgur.com/9Ly0tIM.png
"It would be great if the hacking system allowed to swap the connected wires."
I think this should be worked on being implemented, but sadly don't have the coding knowledge to do it myself so an issue seems most worthwhile.
Some wire crossing ideas.
High Voltage Ground linked to id scan, shocks person trying to open the door if they have a valid or invalid id.
High Voltage Ground linked to bolt wire, when bolts are down, shocks people.
Safety wire linked to bolt wire, crushes people like a safety wire cutting, but bolts down after.
Bolt lights wire linked to power, bolt lights always on, even if door is not bolted.
Open linked to id scan enables emergency access?
Goes without saying wireless transmitter taped onto interface is the best thing ever.
Any other ideas?

I guess you could define it as "each wire can be 'pulsed' such that if it is crosswired, it will act as if the wire it connects to was pulsed"?
So basically the ID lock wire "gets pulsed" when you try to open the airlock, so if you connect that through to bolts it'll pulse the bolts wire.

Game design issues are now being removed from Github and added to the HackMD docs.
Your issue was moved to here.


How to find correct OSD_STAGING_1_OU_BASE_ID

This is needed to have the AAO running and it can differ, depending
where osdctl assigns the account.

Codecov Report

Merging #694 (49db93a) into master (a52a58f) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #694   +/-   ##
=======================================
  Coverage   30.94%   30.94%
=======================================
  Files          48       48
  Lines        5209     5209
=======================================
  Hits         1612     1612
  Misses       3482     3482
  Partials      115      115


Install Datadog APM

Value Statement
As a DevOps Engineer
I want to enable easy and efficient diagnosis of our application
So that actionable tickets are created without time wasted on inefficient diagnosis
Acceptance Criteria
GIVEN we have successfully deployed the snippet into our application
THEN we can see metrics and data in the APM section of the Datadog dashboard
Checklist

[ ] Put the snippet in the correct location in our code base

from ddtrace import tracer

# Network socket
tracer.configure(
  dogstatsd_url="udp://localhost:8125",
)

# Unix domain socket configuration
tracer.configure(
  dogstatsd_url="unix:///var/run/datadog/dsd.socket",
)

[ ] Add the Datadog Tracing library pip install ddtrace to our requirements.txt
[ ] Make sure our pip version is 18.0.0 or greater
[ ] Instrument our application use the included ddtrace-run command. To use it, prefix our Python entry-point command with ddtrace-run.

From their documentation:
For example, if your application is started with python app.py then:

    ddtrace-run python app.py

[ ] Check it's set up correctly by running:

ddtrace-run --info

Assumptions

the Forwarder is set up correctly

Additional Info/Resources

Instrument Your Application

Open Questions

Why would the migrations step cause this to fail?
What tests beyond successful deployment?

Out of Scope / Next Steps

Create a Spike to open up Story tickets to fully utilize Datadog's capabilities and others
We still have yet to fully utilize the logs being forwarder to Datadog, so Spike/tickets should be created in that regard
Do we want APM for the frontend as well?  Real User Monitoring (RUM)
Our glossary and documentation updated

Cross Functional Checklist
Monitoring:

[ ] statsd metrics
[ ] datadog graphs
[ ] datadog alerts

@ldraney would you please add your estimate to this ticket please? Thank you!

Here's my rough-draft approach to solving this problem:
First, try to put the snippet somewhere else besides application.py…
Then, try to run migrations locally and see if my changes are still stopping migrations
Finally, if the above attempts didn't work, I'll make a ticket about “limited migrations errors” and add it as a blocker to my apm ticket
make some artifacts for migrations
modify migrations script and try to get output

When trying to run locally, I forgot to add the package to our application!
migrations_1  | Usage: flask db upgrade [OPTIONS] [REVISION]
migrations_1  |
migrations_1  | Error: While importing "application", an ImportError was raised:
migrations_1  |
migrations_1  | Traceback (most recent call last):
migrations_1  |   File "/usr/local/lib/python3.8/site-packages/flask/cli.py", line 240, in locate_app
migrations_1  |     import(module_name)
migrations_1  |   File "/app/application.py", line 28, in
migrations_1  |     from ddtrace import tracer
migrations_1  | ModuleNotFoundError: No module named 'ddtrace'
migrations_1  |

Migrations passed!  In order to get it to pass we added the ddtrace library to our notification-api/requirements-app.txt  (if this is the right place yet I still need to verify with Kyle).
So, Danny (the customer service at Datadog GovCloud) and I are trying to add the command to our flask app:
ddtrace-run python app.py

In the right place.  We are thinking maybe maybe notification-api/scripts/run_app.sh, or maybe the dockerfile for our app:

Okay, we added the command to run_app.sh, as mentioned above, and are getting an error:
app_1         | ERROR 2023-02-13 19:13:20,434 /usr/local/lib/python3.8/site-packages/ddtrace/internal/writer.py:594: failed to send, dropping 4 traces to Datadog Agent at http://localhost:8126/v0.5/traces after 3 retries ([Errno 99] Address not available)

Which makes sense, because there is no datadog agent locally, I believe.
I'm going to push changes, pass tests, then try deploying to dev and maybe it will work.

ecs deployed successfully, but since I can't ssh into an ecs container, I can't verify that apm is working yet until it gets to our dashboard.
It may already be in the dashboard... I don't see it
so I'm going to begin adding the metric correlation to logs by adding DD_PROFILING_ENABLED=true to the ddtrace-run python app.py command:
ddtrace-run flask run -p 6011 --host=<IP_ADDRESS>

If APM isn't getting to our dashboard in like ten minutes, I'll add this snipped to our application.py:
from ddtrace import patch_all
patch_all()

and then remove the ddtrace-run from the flask run command

In the ECS logs for service "dev-notification-api-service", its says:
2023-02-13 21:00:37 UTC | PROCESS | ERROR | (pkg/forwarder/transaction/transaction.go:357 in internalProcess) | API Key invalid, dropping transaction for https://process.ddog-gov.com/api/v1/container

it's a strange error that's not in the previous logs, so I'll just try re-deploying and see what's up.

Okay, now it successfully deployed!  But, the datadog agent can't communicate with our dashboard!!!
2023-02-13 21:25:41 UTC | CORE | ERROR | (pkg/forwarder/transaction/transaction.go:357 in internalProcess)
 | API Key invalid, dropping transaction for https://7-40-0-app.agent.ddog-gov.com/api/v1/check_run?api_key=eAx(_h3Wut7mIz>{

I can probably just put the API key in the task definition, but the agent should already have it, so this error is weird.

We are still getting this API key error, even after kyle tried updating the agent's api key... so...  I sent a message to Danny:
Hey Danny!
Thanks for the advice.
our agent container is trying to send to our dashboard, but it's getting this error:
2023-02-13 22:43:23 UTC | PROCESS | ERROR | (pkg/forwarder/transaction/transaction.go:357 in internalProcess) | API Key invalid, dropping transaction for https://process.ddog-gov.com/api/v1/container
This is weird because our agent is already sending metric information to our dashboard.
Can you help me with this?
I've tried updating our agents DD_API_KEY, but still the same error.
Where is the agent or APM looking for this key?

Hi Lucas,
The process agent actually does not send any APM data, so I'm leaning more towards a firewall/proxy issue here. Can you make sure all the endpoints listed here have been whitelisted so the agent can connect to them: https://docs.datadoghq.com/agent/guide/network/?tab=agentv6v7#overview?
Can you also try curling the API key validation endpoint and sharing the output: https://docs.datadoghq.com/api/latest/authentication/?code-lang=curl#validate-api-key?
example:
curl -X GET "https://api.ddog-gov.com/api/v1/validate"
-H "Accept: application/json"
-H "DD-API-KEY: ${DD_API_KEY}"
Finally, can you confirm the last 4 characters of the API key you set in the agent config?
Best,
Danny Serrano | Solutions Engineer

It seems to be a network/firewall issue. Lucas received a list of endpoints, to check whether they might be whitelisted. He also verified a key that Danny shared.

The tracer is working correcdtly, but it's not receiving data in ECS:

So I emailed Danny at Datadog Gov, who said:
Hi Lucas,
Thanks for the update! If traces were being sent in, you would see them here: https://vanotify.ddog-gov.com/apm/traces
Continuous Profiling has some additional setup: https://docs.datadoghq.com/profiler/
But let's focus on getting the traces in first.
From the log you shared, it seems the tracer is working correctly, but it's not receiving data: trace agent running
Can you confirm traffic is being generated on the application? Can you also the traces are being sent to the correct port?
Can you also share the following:
your updated task definition
your tracer config (your dockerfile)
confirm your application meets the compatibility requirements here: https://docs.datadoghq.com/tracing/trace_collection/compatibility/python/
send tracer debug logs: https://docs.datadoghq.com/tracing/troubleshooting/tracer_debug_logs/#enable-debug-mode
agent debug logs: set DD_LOG_LEVEL to debug
Best,
Danny Serrano | Solutions Engineer

I'm going to try updating the Datadog Agent on every service in our cluster and see if we start getting logs forwarded.

I also we have not agent in our celery task.  That's interesting.

I'm getting "no data received" on celery as well:

From this ddtrac docs troubleshooting I saw this:
"
Traces not showing up in the app¶
The most common reason for traces not being received by Datadog is an agent communication issue:
Ensure the Datadog agent is running and reachable over the network if not on the same host.
Ensure that ddtrace is configured with the hostname and port of the agent. See Configuration for the configuration variables.
To verify that a connection can be made to the agent with your environment variable configurations run the command ddtrace-run --info.
"

I updating our application.py dd-tracer part to hopefully start looking for the right hostname:
import os
os.environ["DD_TRACE_AGENT_HOSTNAME"] = "my-datadog-agent-hostname"
os.environ["DD_TRACE_AGENT_PORT"] = "8126"  # the default port for trace collection

close

I'm continuiing my conversation with Danny, and will have a screenshare at 3:30 EST today.
I've updated our application and am currently playing with the above configurations to get tracing to our agent (it is still receiving none).
I have noticed that logs are not in our dashboard as they should be (separate issue from a previous ticket), which will be part of the discussion this afternoon.

I'm trying to deploy my changes, but migrations is failing in our pipeline but not locally:

finally got past the migrations issue by merging my branch with the latest master

apm is working :)

And my new code is passing our tests.
Next step is to add the same code to our staging, perf, and prod task definitions, and then I'll open the PR for review.

With Kyle we realized the blocker was my migrations local wasn't being accurate.
I ran a Docker Prune and then rebuilt images on master, then switched to my branch and docker-compose up, and migrations failed locally with the error:
migrations_1  | Usage: flask db upgrade [OPTIONS] [REVISION]
migrations_1  |
migrations_1  | Error: While importing "application", an ImportError was raised:
migrations_1  |
migrations_1  | Traceback (most recent call last):
migrations_1  |   File "/usr/local/lib/python3.8/site-packages/flask/cli.py", line 240, in locate_app
migrations_1  |     __import__(module_name)
migrations_1  |   File "/app/application.py", line 3, in <module>
migrations_1  |     from ddtrace import patch_all
migrations_1  | ModuleNotFoundError: No module named 'ddtrace'
migrations_1  |

This is a good thing as I can finally diagnose what might be going wrong.  In this case, maybe ddtrace==1.8.0 in requirements-app.txt is not the right version...

our current dependency is out of date:
google-cloud-bigquery 3.4.1 depends on packaging<22.0.0dev and >=14.3

latest is google-cloud-bigquery 3.6.0
I'd like to update it as a part of my ticket, because:
ddtrace 1.8.0 depends on packaging>=17.1

But I can also lower ddtrace version, but it seems better to update.

I updated to 3.6.0, and it's passing test, but I'm checking https://pypi.org/project/google-cloud-bigquery/ and the repo to check for potentially breaking changes to our tests.

The next issue is this step is stalling:

First, deploy is now working as normal :)  It is only on a new dev_deploy of my branch that it takes so long.  But on a second deploy it runs like a charm.  May have to do with the task definitions.

I'm getting a weird error from our tests.yaml workflow:
E: The repository 'https://packages.microsoft.com/ubuntu/22.04/prod jammy Release' is no longer signed.
Error: Process completed with exit code 100.
Has anyone seen this before?
The step failing is 'install libcurl' which is running sudo apt-get update && apt-get install -y libssl-dev libcurl4-openssl-dev , but it may have to do with Github Actions using   http://azure.archive.ubuntu.com/ubuntu jammy-security/universe as it's image.

@ldraney I am also seeing this error, I think something broke upstream for many people


From the Modeling tab, choose Sort by Column and then select the State Sort Order field.

From the Modeling tab, choose Sort by Column and then select the State Sort Order field.
Add State Sort Order section, Step4. needs to be changed to From the Column Tools tab (Data view), choose Sort by Column and then select the State Sort Order field.
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 71c046b7-9bc7-2b63-9401-72af0e1a577e
Version Independent ID: d9d85ca9-74ba-63bd-0edf-8f95754d4991
Content: Add a time-in-state measure to a Power BI report - Azure DevOps
Content Source: docs/report/powerbi/create-timeinstate-report.md
Product: devops
Technology: devops-analytics
GitHub Login: @mijacobs
Microsoft Alias: kaelli

@farinhate - Thank you for the feedback. I'll update the article later today.


UPSTREAM: : conditionally rename manifests

Description

Reviewer Checklist

[ ] API Go Documentation
[ ] Tests: Unit Tests (and E2E Tests, if appropriate)
[ ] Comprehensive Commit Messages
[ ] Links to related GitHub Issue(s)

/approve

Weird that my /approve didn't take. The OWNERS files exist here and the release repo config merged a few days ago...
I'll manually label it.


stacking

Hi Matt, another submission, tried stacking this time!

Hello! Thanks for this Wouter. If I'm doing this right (taking the arrays from the ends of the two notebooks), this is scoring 0.535. Cheers!

How cruel :( This approach gave me over 0.66 when taking churchman bible as blind.. I guess I'm still overfitting this..I'll be back..

I think we're all learning that earth is a cruel mistress. Or at least a really unpredictable one.


Glossary terms v2

After a short workathon with @Chalarangelo we are finishing up Glossary terms for #690. Refer to commit history and files for terms added.
What does your PR belong to?

[ ] Website
[ ] Snippets
[x] General / Things regarding the repository (like CI Integration)
[ ] Tests

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (non-breaking improvement of a snippet)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.
[x] I have checked that the changes are working properly
[x] I have checked that there isn't any PR doing the same
[x] I have read the CONTRIBUTING document.

Thanks, @Chalarangelo - I've fixed mentioned things, hopefully, everything is ready for merging now.


Add logging configuration from file.

Codecov Report

Merging #15 (c390c4d) into master (d6e6014) will increase coverage by 1.00%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master      #15      +/-   ##
==========================================
+ Coverage   91.30%   92.30%   +1.00%
==========================================
  Files           5        5
  Lines          46       52       +6
==========================================
+ Hits           42       48       +6
  Misses          4        4

Flag
Coverage Δ

unittests
92.30% <100.00%> (+1.00%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

fingerprint/entrypoints/cli/main.py
93.75% <100.00%> (ø)

fingerprint/service_layer/handlers.py
90.90% <100.00%> (+2.02%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d6e6014...c390c4d. Read the comment docs.