Create global and local allocations

The goal is to create local allocations which could be a reference to the global allocation for each tile to access the memory. The prototype is as below
We leave the memref.alloc op as a global variable:
%alloc = memref.alloc() : memref<1x1x8x4x8x4xi32, 2 : i32>
And create a new pointer pointing to the global allocation as
%ptr = amdaie.addressof %alloc : memref<1x1x8x4x8x4xi32, 2 : i32> -> !amdaie.ptr
To load the memory:
%local_alloc = amdaie.load %ptr : !amdaie.ptr -> memref<1x1x8x4x8x4xi32, 2 : i32>
And then replace all uses of %alloc with %local_alloc.
@MaheshRavishankar Could you take a look and see if I understand this correctly?  And do I miss anything?
@jtuyls It would be helpful if you could add some context of how we are going to use the local allocations on different tiles.

So we dont technically need a "global variable". We can leave the alloc as it is today, just move it to the top of the function.

And create a new pointer pointing to the global allocation as
%ptr = amdaie.addressof %alloc : memref<1x1x8x4x8x4xi32, 2 : i32> -> !amdaie.ptr

you dont need a new type. You can just use
%memref_ref = amdaie.reference_to %alloc : memref<..., 2: i32>

%memref_ref is the same type as %alloc, but the reference is not hoistable across the scf.forall boundary. This will make the amdaie.load isntruction unnecessary. You just have to replace all %alloc with %memref_ref.


Expanding connections (et al) crashes on Mac Sonoma 14.3

Referenced @buliverd53 in another thread, but now spawning separate:
I've discovered that the problem happens when I click on the expand arrow on the left side of the name of the connection:

every time i click this arrow, the app crashes. If i just double click to connect, doesn't crash, but if I try to expand any other view, like DataBase, crashes the same way.
I've attached the debug log.
dbeaver-debug.log
Originally posted by @buliverd53 in https://github.com/dbeaver/dbeaver/issues/22116#issuecomment-1847396187
Confirming the same behaviour here (both in brew cask and ce.dmg).

hs_err_pid82696.log
hs_err_pid82561.log

Hi thank you for report it looks like an issue with graphics on macOS, it can quite possibly can be connected to changes in macOS Sonoma 14.3.

macOs Sonoma 14.3 should have a stable release soon. If the issue persists, we will contact Apple.

This seems to be resolved in 14.3 beta 2

Confirmed here as well.
Resolved in 14.3b2 with <IP_ADDRESS>312241705


Continuous integration : configure travis CI support

Add travis.yml file
Enable CI

Build fixed and status image shown


Found a faster way to download mods from wabbajack

Hello Im not good in english nor do i have a good knowledge in coding so sorry if my explanation/request is hard to understand but I'll try my best.
The software has wabbajack files for each modlist. There is 'modlist' file inside this wabbajack file that can be extracted using zip extractor like 7z. img:
This file contains thousands of lines of texts containing every mod with its details, modID and fileID. I was thinking of a code that could extract each modID and fileID found in the 'modlist' file with some kind of wildcard pattern and turn it into download link and remove all other unnecessary string/characters so it would be only: https ://www.nexusmods.com/skyrimspecialedition/mods/[modID]?tab=files&file_id=[fileID] .
So for wildlander modlist, some of the lines in the modlist file like below

can be turned to

Spoiler
https://www.nexusmods.com/skyrimspecialedition/mods/17925?tab=files&file_id=177264
https://www.nexusmods.com/skyrimspecialedition/mods/43097?tab=files&file_id=204112
https://www.nexusmods.com/skyrimspecialedition/mods/13366?tab=files&file_id=114102

that means if the modlist has 500 mods, the extracted data should be 500 of the download links in the format above. Now I'm requesting for a code if its possible that would just to automatically open up all these links in a browser. The reason is Wabbajack cannot have extension installed in their built in browser(edge webview2) and I could lets say download half of the links with my static internet IP address on 1st instance of chrome and use an vpn extension like urbanvpn on 2nd instance of chrome to download other half download links bypassing nexusmods download limit all without the 5 second wait thanks to this userscript: https://greasyfork.org/en/scripts/394039-nexus-no-wait.
After downloading all the mods, opening wabbajack would be just to verify and finish installation without much hassle :D
really love ur autoDL but with these methods, downloading would be several times faster!
edit: or you could just implement a code in the current autoDL that injects the 5 seconds skip userscript into WJ if its possible

Courtesy of chatgpt
import json
Read the JSON file
with open('modlist', 'r') as file:
json_data = file.read()
Parse the JSON data
try:
data = json.loads(json_data)
except json.JSONDecodeError as e:
print(f"Error: Failed to decode JSON - {str(e)}")
exit()
Create a list to store the generated URLs
urls = []
Process each entry and create URLs
entries = data["Archives"]
for entry in entries:
print(entry)
try:
mod_id = entry['State']['ModID']
file_id = entry['State']['FileID']
url = f"https://www.nexusmods.com/skyrimspecialedition/mods/{mod_id}?tab=files&file_id={file_id}"
urls.append(url)
print(url)
except TypeError:
print("Error: Invalid entry format - skipping entry")
except KeyError:
print("Error: Missing required keys in entry - skipping entry")
Write the URLs to an output file
with open('output.txt', 'w') as file:
for url in urls:
file.write(url + '\n')

Also courtesy of ChatGPT. This opens 20 of the download links at a time. Let them finish downloading and press enter in the terminal for the next 20.
`import webbrowser
Read the links from the text file
with open('output.txt', 'r') as file:
links = [line.strip() for line in file.readlines()]
Open links in batches of 20
batch_size = 20
current_batch = 0
total_links = len(links)
while current_batch < total_links:
# Determine the range of links to open in this batch
start_index = current_batch
end_index = min(current_batch + batch_size, total_links)
batch_links = links[start_index:end_index]
# Open each link in the batch
for link in batch_links:
    webbrowser.open(link)

# Wait for keyboard input before opening the next batch
input("Press Enter to continue to the next batch...")

# Move to the next batch
current_batch += batch_size`

I have made a tutorial repo.


Conditional snippets that don't satisfy conditions still appear in cmp menu

Wanted to see if there was some VimTeX syntax issue I was missing when setting environment snippets, turns out I found something interesting with LuaSnip.
From this VimTeX issue we have the following code which successfully produces a contextual snippet.
-- File: nvim/LuaSnip/tex.lua
-- Loaded using the LuaSnip Lua loader (`:help luasnip-lua`)

local tex = {}
tex.in_tikz = function()
  local is_inside = vim.fn['vimtex#env#is_inside']("tikzpicture")
  -- perhaps using both tests is redundant and only the is_inside[1] is needed?
  return (is_inside[1] > 0 and is_inside[2] > 0)
end
-- Not needed in the context of this question but perhaps useful
tex.in_mathzone = function() return vim.fn['vimtex#syntax#in_mathzone']() == 1 end
tex.in_text = function() return not tex.in_mathzone() end

-- Return snippets
return
{
  s({trig="tikztest"},
    {
      t("Works!"),
    },
    {condition = tex.in_tikz}
  ),
}

However, on completion menu the trigger still appears despite not fitting the condition, which confused me at first:

Is there a way to fix this?

Ah, condition is only queried on manual expansion (via tab, for example).
To hide the snippets in cmps menu, you can use show_condition :)
(you can pass functions written as show_condition to condition;, but not the other way round (cmp doesn't have the data for condition` readily available, and would have to compute it first, which doesn't seem worth it, hence the separation))

I am running into a somewhat similar problem but I cannot manage to get the syntax right for show_condition: in particular I am trying to write a condition that only shows the snippet if a line is empty (simulating the beginning of line):
show_condition = function()
  return vim.api.nvim_get_current_line() ~= ""
end,

doesn't seem to work, and the documentation for it indicates:

show_condition: f(line_to_cursor) -> bool.

where it's unclear if f refers to a function node alias or to a function definition.
What would be the correct grammar to write a show_condition for empty lines?

Oh, the doc bit misleading, true: it should be a function, not a functionNode.
What isn't working about you example? Looks correct to me

What isn't working about you example?

The condition isn't being evaluated and applied; interestingly, even the built-in doesn't seem to have effect. Must one specify show_condition together with condition or is the former sufficient? I tried to browse the examples file but couldn't find any relevant example.

Could you open a new issue and upload a video there?

I will do: I will double check that all the rest in my configuration is fine and prepare a minimally reproducible example.


Add drug field to gene and variant browse

As long as it's not too squished we would like to add a drug list like the disease list to the variant/gene browse tables.

An additional Drugs column on those will just take a few minutes. @acoffman, does adding the Drugs attribute to those endpoints present any issues?


Following inletsctl create docs for Azure fails with authorization errors.

The reference docs for AWS and GCE both contain detailed instructions for setting up the authorization needed to run inletsctl create. The corresponding docs for Azure are incomplete and following them doesn't work without an extra step.
Expected Behaviour

Install inletsctl
Follow the steps listed here: https://docs.inlets.dev/reference/inletsctl/#example-usage-with-azure
Inlets is deployed successfully

Current Behaviour

az ad sp create-for-rbac --sdk-auth > ~/Downloads/client_credentials.json prints the following but succeeds anyway:

WARNING: Option '--sdk-auth' has been deprecated and will be removed in a future release.

The inletsctl create command shown in the docs  (with my subscription id instead of the one in the docs) fails with the following output:

Using provider: azure
Requesting host: elegant-sammet6 in eastus, from azure
2022/01/30 15:40:01 Provisioning host with Azure
2022/01/30 15:40:01 Creating resource group inlets-elegant-sammet6
resources.GroupsClient#CreateOrUpdate: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailed" Message="The client '<service-principal-object-id>' with object id '<service-principal-object-id>' does not have authorization to perform action 'Microsoft.Resources/subscriptions/resourcegroups/write' over scope '/subscriptions/<my subscription id>/resourcegroups/
inlets-elegant-sammet6' or the scope is invalid. If access was recently granted, please refresh your credentials."

Possible Solution
The service principal needs to be granted the appropriate authorization to be able to create the resources needed. This is clearly overkill, but assigning the service principle the 'Owner' role at the subscription level works:
az ad sp create-for-rbac -n inletsctl --sdk-auth > az-sp.json
CLIENT_ID=$(jq -r '.clientId' az-sp.json)
SUBSCRIPTION_ID=$(jq -r '.subscriptionId' az-sp.json)
az role assignment create --assignee $CLIENT_ID --role "Owner" --subscription $SUBSCRIPTION_ID
inletsctl create --provider=azure --subscription-id=$SUBSCRIPTION_ID --region=australiaeast --access-token-file=az-sp.json

It should be possible to narrow down the actual permissions required to run the deployment.
Your Environment

inlets PRO version: 0.9.3 - 8e96997499ae53c6fb2ae9f9e13fa9b48dcb6514
inletsctl version: 0.8.14 - 4034ead80df9bd487d18ca9fd1af4856ab7a9e71
Cloud provider and region being used: Azure, eastus and australiaeast
Operating System and version (e.g. Linux, Windows, MacOS): WSL2

Hi @eltariel thanks for mentioning this.
We don't see to have a record of you as a customer. Sorry if I am not connecting the dots.

It should be possible to narrow down the actual permissions required to run the deployment.

This was working when it was contributed by the community, but I don't personally use Azure so haven't tested it recently. We would be open to a contribution if you can come up with an alternative for the documentation.
Alex

Hi @eltariel thanks for mentioning this.
We don't see to have a record of you as a customer. Sorry if I am not connecting the dots.

No worries, I'm not actually a customer yet. I've been evaluating options for my little pi cluster (your guides have been super helpful with getting that going!) and inlets looks like a winner.

It should be possible to narrow down the actual permissions required to run the deployment.

This was working when it was contributed by the community, but I don't personally use Azure (often) so haven't tested it recently. We would be open to a contribution if you can come up with an alternative for the documentation.
Alex

I'll have a look when I get a chance and see if I can find a more precise set of permissions. I'm still fairly new to azure myself, I just happened to have some free capacity available.

The feedback is appreciated, however I should make it clear that you do need a valid license or Gumroad subscription to deploy, launch or use inlets-pro in any way.
inletsctl is really a utility method for creating VMs and preinstalling the server component. It's optional, and shouldn't affect your Pi cluster. The same goes for using it in-cluster, where Helm may be preferable over the operator for a tunnel for ingress: https://inlets.dev/blog/2021/12/06/expose-traefik.html
Alex

The feedback is appreciated, however I should make it clear that you do need a valid license or Gumroad subscription to deploy, launch or use inlets-pro in any way.
inletsctl is really a utility method for creating VMs and preinstalling the server component. It's optional, and shouldn't affect your Pi cluster. The same goes for using it in-cluster, where Helm may be preferable over the operator for a tunnel for ingress: https://inlets.dev/blog/2021/12/06/expose-traefik.html
Alex

Thanks for the comments. My understanding is that both inletsctl and inlets-operator use the same method for deploying the vm, and I was mostly checking that would work before I tried to figure out how to use inlets-operator. Exposing services on my cluster isn't the only use case I have for inlets 😄
Just want to clarify that I haven't attempted to actually use inlets-pro yet, just verifying that the vm creation was as straightforward as it appeared. I had intended to grab a licence before installing the local end of the tunnel, and the vm was only live for long enough to validate that it created correctly. Apologies for misinterpreting the licence, I was under the impression that I would be fine to test this functionality as the licence appears to be tied to the client rather than the server. (Please note this isn't meant to be sarcastic, I can't work out any other way to phrase it though)

Personal licenses are priced at a trivial amount of money per month, so that you can trial them on a month by month basis. If you like inlets, you can upgrade to annual pricing for an additional discount on the costs.
The license is enforced on the client so as not to make server tunnel maintenance a thing, but the whole application is subject to the EULA and requires a valid license.
The product actually works as advertised and has dozens of active commercial and personal users, so I would encourage you to purchase and commit to it. Or find a free alternative if that suits your needs better. There will be trade-offs whatever you do. I talk a bit more about that here -> https://inlets.dev/blog/2021/07/27/monthly-subscription.html
And here -> https://inlets.dev/blog/2021/12/31/inlets-2021-highlights.html
If you want ingress for your K8s cluster, I prefer this method -> https://inlets.dev/blog/2021/07/08/short-lived-clusters.html


Cannot read property 'location' of null when use useRoutes

Version
6.0.0-alpha.3
Test Case
https://codesandbox.io/s/frosty-panini-ifqcj?file=/src/App.js
Steps to reproduce

Open the codesandbox project
You'll see an error is thrown straight away: Cannot read property 'location' of null

Expected Behavior
I'd expect "home" to be rendered in the page
Actual Behavior
An error, Cannot read property 'location' of null, is thrown

You have to wrap it with a Router, preferably at the root of the tree.

Hi, how you solved this case? Same issue here.

@timdorr I have a BrowserRouter containing the routes. But still throws the error. Seems that can't create a context.
export const App = () => {
  const routes = useRoutes(appRoutes);

  return (
    <Provider store={store}>
      <BrowserRouter>
        <aside>
          Menu lateral
        </aside>
        { routes }
      </BrowserRouter>
    </Provider>
  );
};

You're still using the hook before the router component. You have to have it above that component or move the useRoutes down the tree.

@timdorr I got it, just put the useRoutes inside a child componente of App and it works. Thank you!


DBZ-8386 Consume all records before starting actual snapshot test

This seems to be more reliable then waiting more or less random time and assuming that all transaction were already processed by the database.
https://issues.redhat.com/browse/DBZ-8386

DB2 fails with
Error:  Errors:
Error:    Db2ConnectionIT.shouldEnableCdcForDatabase:24 » SQL ASNCAP server did not start.

which is unrelated.


feat: add Multikey support

What is being changed
This PR upgrades did-jwt, did-jwt-vc and @aviarytech/did-peer dependencies.
The update fixes issues with did:peer identifiers that use multiple service endpoints as well as adding support for Multikey verification methods.
The updated libs also allowed the simplification of some codepaths that involved checking the key type and the fixing of some corner cases where the wrong key types were mistakenly used (like using secp256k1 for encryption).
Quality
Check all that apply:

[X] I want these changes to be integrated
[X] I successfully ran pnpm i, pnpm build, pnpm test, pnpm test:browser locally.
[X] I allow my PR to be updated by the reviewers (to speed up the review process).
[ ] I added unit tests.
[ ] I added integration tests.
[X] I did not add automated tests because _________, and I am aware that a PR without tests will likely get rejected.

Details
See:

https://www.w3.org/TR/vc-data-integrity/#multikey
https://identity.foundation/peer-did-method-spec/#resolving-a-didpeer2
https://w3c-ccg.github.io/did-method-key/#document-creation-algorithm

Codecov Report
Attention: 12 lines in your changes are missing coverage. Please review.

Comparison is base (c050588) 85.47% compared to head (40ad8ef) 85.50%.

Files
Patch %
Lines

packages/utils/src/did-utils.ts
65.51%
10 Missing :warning:

packages/utils/src/jwk-did-utils.ts
0.00%
2 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             next    #1316      +/-   ##
==========================================
+ Coverage   85.47%   85.50%   +0.03%
==========================================
  Files         170      170
  Lines       18952    18946       -6
  Branches     2116     2115       -1
==========================================
+ Hits        16199    16200       +1
+ Misses       2753     2746       -7

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


More Rust 2018 edition cleanups

@bors: r+

:pushpin: Commit 54c42142517cfe5b29e8d155fa40a6525d0abd54 has been approved by alexcrichton

:hourglass: Testing commit 54c42142517cfe5b29e8d155fa40a6525d0abd54 with merge 5bb302352a65d4cdcb7d0fe30bcb1a921b7ae8e3...

:sunny: Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing 5bb302352a65d4cdcb7d0fe30bcb1a921b7ae8e3 to master...


singularity pull rate limit

Error run into by Ayslan on biowulf the first time he tried to run RENEE:
Failed to pull singularity image from docker://nciccbr/ccbr_rseqc_4.0.0:v1.0:
FATAL:   While making image from oci registry: error fetching image to cache: failed to get checksum for docker://nciccbr/ccbr_rseqc_4.0.0:v1.0: reading manifest v1.0 in docker.io/nciccbr/ccbr_rseqc_4.0.0: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Implementing the shared SIF cache dir by default should help so users don't have to pull containers already pulled by others.

Ultimately the problem was the user was calling renee run once per fastq file instead of passing all fastq files at once. I could see how users may hit the dockerhub rate limit though if trying to run renee for multiple projects simultaneously.

If other users run into legitimate rate limit issues due to running renee simultaneously on multiple data sets (rather than a user incorrectly running once per sample), we may want to look into how users can authenticate their dockerhub account on biowulf to increase their rate limit and add instructions to our docs. Until then, this is likely not a true issue.


feat(kit): get module options with getNuxtModuleOptions

🔗 Linked issue

❓ Type of change

[ ] 📖 Documentation (updates to the documentation, readme or JSdoc annotations)
[ ] 🐞 Bug fix (a non-breaking change that fixes an issue)
[ ] 👌 Enhancement (improving an existing functionality like performance)
[x] ✨ New feature (a non-breaking change that adds functionality)
[ ] 🧹 Chore (updates to the build process or auxiliary tools and libraries)
[ ] ⚠ Breaking change (fix or feature that would cause existing functionality to change)

📚 Description

When integrating with other modules, it's common to use some of their configuration. Relying on just the module config key isn't safe, instead, we introduce a utility to extract the options in a safer way.
For example, within Nuxt Simple Sitemap:
if (hasModule('@nuxtjs/i18n')) {
  const i18nVersion = await getNuxtModuleVersion('@nuxtjs/i18n')
  if (!await hasNuxtModuleCompatibility('@nuxtjs/i18n', '>=8'))
    logger.warn(`You are using @nuxtjs/i18n v${i18nVersion}. For the the best compatibility, please upgrade to @nuxtjs/i18n v8.0.0 or higher.`)
  const nuxtI18nConfig = await getNuxtModuleOptions('@nuxtjs/i18n') as NuxtI18nOptions
  if (nuxtI18nConfig?.pages) {
    // do logic
  }
}

📝 Checklist

[ ] I have linked an issue or discussion.
[ ] I have updated the documentation accordingly.

This PR should also look to investigate if it's feasible to use this function internally to resolve the options for a module, so that we know the logic is consistent. This would also let us implement a caching layer to reduce the time needed to setup modules.

Nice! I'll track this PR to include it in new kit documentation after merge

@harlan-zw Is this still something you are planning on working on?

I don't have the capacity to see if we can use this function internally, so we can either close this, merge it as is, or someone else wants to take it on.


fix(docs): Update BUILD.md linux instructions to use correct clang ve…

…rsion
changes clang*17 to clang*18

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Method getAllJobsForTag all time return 0 on Android 4

Hi guys!
I using the latest version 1.1.6 and have some problem while checking my jobs.
I run this code for checking already scheduled jobs:
int jobsSize = JobManager.instance().getAllJobsForTag(MY_JOB_TAG).size()
Speaking about android OS 5, 6, 7 this works perfectly and jobsSize doesn`t equals 0.
but on android OS 4 jobsSize all time is 0.
Could you help me?
Thanks

Are you sure that you want to get all jobs? That returns only jobs which are currently running or just finished. Whereas JobManager.instance().getAllJobRequestsForTag("TAG") returns all pending jobs.

Hi @vRallev,
Thanks for your reply.
Yes it`s strongly right. Could you chek this on your android 4 os device?

I did and it works for me. Note that once your process died the jobs are cleared as there's no point to reload already finished jobs. Please attach logs that show when your job ran and when you're querying the jobs. Also make sure that your job doesn't crash or that the process died in between.

Any update?

Closing because of inactivity


Coach Role

Coach features:
Interact with (training-) plan:

[x] create
[x] edit
[x] load from store
[x] save to store
[x] view
[x] delete
[x] assign score

Done, but we still have to create roles different from coach


Mod doesn't appear in the mod menu, nor do any items appear, despite it being in the mods folder. it's like it doesn't exsist

Which mod loader was used?

forge

The mod is only for Quilt (Fabric on older versions of the mod). Forge will not load it.


Support Visual Studio 2015

With the Visual Studio 2015 RC being released recently and the pending release of VS2015, it would be nice to have 2015 support.

+1

+1

+1

Doing so,e research, the RC of the 2015 SDK is available at https://www.microsoft.com/en-us/download/details.aspx?id=46850
I may look into what it would take to get a functioning 2015 build beside the 2013 build, to have something ready for the full release of 2015.

:+1:

+1

+1

:+1:

Look, just adding +1's clutters up the thread and adds absolutely no value. It's a desirable feature. Lots of people want it for obvious reasons. You don't need to chime in.

@cmr :+1: on your comment regarding the +1's.

This should be fixed by #138

Please don't forget to update/add the extension at the Visual Studio Gallery.

Yes, I'm just waiting to resolution of a problem with AppVeyor

Once we get few small things done (debug settings, etc.). I'll put 0.1.1 with 2015 support. meanwhile VS 2015 support is available in CI builds.


Change dbus interfacing for making rootless podman containers work

ref : https://github.com/containers/youki/issues/719 , https://github.com/containers/youki/pull/1171
check https://github.com/containers/youki/pull/1171#issuecomment-1387207719 , https://github.com/containers/youki/pull/1171#issuecomment-1586927926 first.
This is a summarized version of discussion utam0k, yihuaf and me had on discord regarding the changes needed for making youki work with podman rootless.

For youki to be able to run in rootless way via podman, we need to (atleast)  change the way we have implemented dbus interface. The reason for this is as follows :
When running under podman rootless, we need to connect to a socket at /run/user/(uid)/bus and then authenticate this socket to dbus using some kind of authentication mechanism. We need to access this socket path as root (because this is at /run, and thus would be a protected path(?)); however, when we authenticate with dbus, we must use the actual user id that has started the (non-root) podman process (i.e. 1000 etc)
Currently we use dbus-rs, which exposes rust binding to libdbus via ffi calls. Now libdbus provides one function to open a socket at given path, and another function to register the connection (which does all the authentication etc.) This creates a problem for us because we need to do both of these things using two different uids. (connect with uid 0, authenticate with actual uid) . Because libdbus (not dbus-rs) finds the effective uid (euid) internally , and then somehow stores euid across these two calls (probably storing it in a structure along with socket ; but not sure) we cannot change the it by doing seteuid call between these two steps.
On a side note, our current code does a Connection::session() https://github.com/containers/youki/blob/main/crates/libcgroups/src/systemd/manager.rs#L188  which is incorrect because it connects to the "default" dbus session bus, and we want to connect on the /run/user/(uid)/bus path. What runc does is that it uses two different libraries in go, one to connect to a socket and other to perform authentication. Both of these libraries have co-operating data-structures, so this can be done there.
The three ways to solve this are :

We don't support running rootless under podman :( We support rootful mode, but not rootless
We own everything about dbus communication in youki. That means we completely remove dbus-rs as a dependency, and implement manually the connection, authentication and everything else. Again, not the best way, because there will need to be a lot of code, we will need to be sure that it works correctly, and we will need to write a lot of tests to validate that.
We try switching to zbus. zbus is an alternate dbus library, but instead of ffi, it implements the authentication natively. Now, this still doesn't let us do the connection and the authentication separately, but I think we can ask them if we can add this method, and in worst case, fork and add code that we need ; keep maintaining it. This is again not a good solution, because maintaining a fork is a difficult task ; even apart from that we have to make a lot of changes to switch from dbus to zbus, as this file is generated specifically for dbus.

* if * we choose to move to zbus, I think a more sensible approach would be to first test in a branch if it will allow us to do what we want (separate connection and authentication) and then first move from dbus to zbus in one pr, and add the custom auth in separate pr. Still a lot of work, but ok.
All in all, none of the options I can think of are particularly good, so want to hear your opinion on them first. Feel free to ask how I found of things I have listed above if you think I might be wrong or missed something or have a better way.
Some Related links :
https://github.com/containers/youki/pull/1171#issuecomment-1586927926
https://github.com/diwic/dbus-rs/issues/443
https://github.com/opencontainers/runc/blob/main/libcontainer/cgroups/systemd/user.go#L21

I have been working on switching from dbus to zbus, and checking if youki can run rootless via podman.
Some caveats :

zbus is pretty bulky and the release compiled binary has an increase in size from 8.5 MB (dbus-rs) to 11 MB zbus)
zbus is async first, and provides sync support that is just a wrapper over async
The default feature async in zbus uses asyncio, which spawns threads in background. This is not acceptable to us, as we need to do unshare(userns) call , and that requires the calling process to not have threads. To make it work, we must disable default features and enable tokio feature, which works without spawning threads.

There are few other changes needed in non-dbus related code as well in order to correctly run youki rootless.
I have made a PR on my fork for

switch from dbus to zbus : https://github.com/YJDoc2/youki/pull/256
switch to zbus and changes for rootless [WIP] https://github.com/YJDoc2/youki/pull/257

cc @utam0k @yihuaf

@YJDoc2 Thank you for all the great work in putting this together and this is great!

The default feature async in zbus uses asyncio, which spawns threads in background. This is not acceptable to us, as we need to do unshare(userns) call , and that requires the calling process to not have threads.

I agree. In addition to the reason you stated, there are also a lot of undefined behaviors around forking a multi-threaded process without immediately exec into something else, which we have to do in youki.

The three ways to solve

I believe there is also a 4th way. We can implement just enough of dbus related logic in youki without forking zbus. We can look to zbus for ideas and borrow code, but we also don't need the entire zbus implementation. Our usecase for this is strictly around the dbus and cgroups. So the end result should be much simpler, and shedding 8+MB for binary size along the way. This would be a great improvement for us. In addition, getting rid of libdbus dependency by itself is a worthy optimization in my opinion.

Since it is clear we want to solve this issue one way or another (supporting rootless under podman is a very important requirement), I would suggest we can do the development in the main repo. We can wrap the experimentation code into a feature and don't compile by default, until we are ready. In this way, we don't have to support everything in a single big PR merge.
There are a few things I think will be helpful:

Fix the bug where we always connect to the default dbus session for rootful mode first. We should have a different issue tracking this.
Fix any other minor issue until we can reproduce the bug mentioned in this issue with the podman test.
Refactor the systemd related code to allow for a new feature that uses an alternative code path. I have not looked at the code, but I think you mentioned connect and authenticate being two places where we need to implement.

Feel free to break this into parts that can be worked on in parallel and let us know how we can help.

Sounds like a good plan :)

@utam0k can you help me with understanding something? In our current code, we have a Rootless structure which we use. The conditions for constructing that here check that if a user namespace is given in spec and its path is None . However, in the rootless_required function here we only check if the current ID of user is 0 or not . If I'm correct, these conditions are not exactly same, as if we are root user, the spec might still contain the user namespace ; and if we are non root user , we must create rootless container even if spec does not have user ns (of course, we must check that in such case we already are in a user namesapce) .
So what I am confused about it iis the purpose of Rootless structure and its naming a bit misleading? Should it be named NewUserNs or something similar? Can you help me understanding this better? Thanks!

If I'm correct, these conditions are not exactly same, as if we are root user, the spec might still contain the user namespace ; and if we are non root user , we must create rootless container even if spec does not have user ns (of course, we must check that in such case we already are in a user namespace) .

Since I think you are right, NewUserNs or something sounds more accurate 👍

@utam0k @yihuaf can I ask you to take a look at https://github.com/YJDoc2/dbus_native ? This does all the implementation of dbus interfacing code that is needed by youki in Rust native, without needing libdbus or other external dependencies (except nix crate). I'm planning to copy that code into youki and replace dbus-rs, and wanted to have it checked a bit before I make the PR. The code is almost final, except changing the Error type to youki-error compatible once I copy it over. There is also one small unsafe which covert string from a vec directly because we know its a valid utf-8 , but I can change that to safe version + unwrap when I copy over.
As a Note this does not implement the complete dbus functionality, just the part that is needed by youki. I have also tried to keep the proxy interface same, so we have minimal changes is our current client implementation.
Thanks :)

@YJDoc2 🎉 🎉 I'll take a look at your repository 👀

@YJDoc2 The overview of its design looks good to me. Let's start to develop in youki repository. By starting development in the youki repository, even if it is not fully worked,  review, etc, can be done in smaller cycles.

I agree with @utam0k. This is more than good enough to move into the youki repo and start to be developed. We can lock this behind a feature initially.

Thank you @utam0k @yihuaf for taking a look! Once #2279 is updated and merged, I'll open a PR which will move and integrate this code into youki.
Also, I was thinking that as long as all current CI and tests pass, we can switch from dbus-rs to this implementation ; however, if you want to lock it behind a feature, that seems fine too. Can you tell me what other conditions/checks we will use for eventually moving this from a feature to default in future, so I can keep that in mind when moving the code? Thanks!

Can you tell me what other conditions/checks we will use for eventually moving this from a feature to default in future, so I can keep that in mind when moving the code? Thanks!

Why don't you ask @orimanabu to verify it, and if it's not a problem, then become default it?

Completed via #2370 :tada:


Add section about Producer configs

Are these fields configurable in the connector? If yes, we should also update in the connector section.

not yet

Can you add a section on how various config settings impacts performance?


DevOps Build Process Optimizations

Today may have been a bit unusual, with an error from solr seeming to repeatedly happen, while meanwhile the Development solr database appears to be updated, as does the Development RDS database.  Maybe I'm missing something.
But it's not acceptable that a full build is still running, 20 hours later.  I ran the entire process here in about 4 hours on my PC.  The worst part of this is that if I were really building a production set with a deadline, this would really extend the time...especially if I had to run it more than once.  Even incremental builds don't solve the problem...I don't have timing data, but as I recall the last time I manually ran one, the incremental process takes many hours as well...when in fact, it may only be a handful of files being loaded.
We need to do better.
Some ideas for improving build performance:

In a rebuild scenario, once we see the crossing point of the forward and reverse runs, we should stop or at least let then ext step (build images) proceed...that takes far too long on AWS to finish even though the work is actually done.  And the fact that the process can't "complete" until each direction has looked at all the files, makes me wait hours more.

So I've added a flag: --halfway which can be used for both the forward and reverse processes--and they will quit after each processing half of the files.  Hopefully, that will help.

We could add more processes to the full build...my 4 hour timing is running forward, reverse, and targeting some of the bigger journals separately (only works when resetting the cores, so it's not set to "all" but rather checks if each file is in the database, so overlap with two main processes isn't a problem, and they just zip past when they are already processed (of course it doesn't seem to zip on AWS).  I sometimes run as many as 6 or so processes here on my Intel I9 9900K 8-core PC to optimize the timing.

For smaller changes it may be a good idea to make another processing option...where I can make more targeted changes to development solr and development RDS quickly...and then generate what's needed for stage.  I can run my builds remotely on those, and for smaller updates, I think it may run more quickly than the current update process.  Then we'd have a script to create what's needed to create the AMI's to push to stage and eventually production (after testing).  This could even run off of stage, as an option...so after checking stage, if there are some issues, it can be dealt with there, then the AMIs generated and pushed to production.

Though we can discuss why incremental also seems to takes so long before going down that road.  One thing that's odd--I noticed when running incremental builds, it's falsely detecting changes and processing some items, where it shouldn't have to.  Not sure why that happens--it doesn't happen when run locally.  Perhaps something about comparing times between S3 and Solr?
I do have another ace up my sleeve to make it faster...but I'm hesitant because it involves a loss of an interesting search feature (even though the client isn't using it) and requires changes to the server as well.  Still I'm considering it if we can't speed things up.  I'm open to ideas.  I am ok with a 12 hour total for a full build--one I can start in the evening and have it ready in the morning.  I'd prefer faster but I can live with that.  Incremental builds should take very little time, anywhere from 15 minutes at most for a few new files to an hour or two, for example, to rerun the whole PEPCurrent folder (<7000 files).
But I don't think a full build that takes over 20 hours, like the current one is taking, works for starters.  Maybe that's an anomaly, but it's certainly holding me up today...and I don't want to stop it to try again, because it seems to take 14 or more hours to rebuild in any case.

I started a build with the --halfway mark last night.  This one also included the new article_tracker data table in RDS which records what article IDs have been seen, so the notify file can be generated at the end of the build (notifying of any new issues/articles).
Each process seems to have run 22K records, and then failed, meaning the database and solr was updated with 44K of the 136K records.  The error is strange...suddenly, after storing 44K records total, running about 6 hours, the ocd database no longer existed in Python's memory, so the commit failed:
  | 2021-02-03T07:13:30.596-05:00 | Processing started at (Wed Feb 3 05:32:06 2021).
-- | -- | --
  | 2021-02-03T07:13:30.596-05:00 | --------------------------------------------------------------------------------
  | 2021-02-03T07:13:30.596-05:00 | halfway option selected. Processing approximately one-half of the files that match.
  | 2021-02-03T07:13:30.596-05:00 | Load process started (Wed Feb 3 05:32:06 2021). Examining files.
  | 2021-02-03T07:13:30.596-05:00 | Committing info for 1000 documents/articles
  | 2021-02-03T07:13:30.596-05:00 | Committing info for 1000 documents/articles
  | 2021-02-03T07:13:30.596-05:00 | Committing info for 1000 documents/articles
  | 2021-02-03T07:13:30.596-05:00 | Committing info for 1000 documents/articles
  | 2021-02-03T07:13:30.596-05:00 | Committing info for 1000 documents/articles
  | 2021-02-03T07:13:30.596-05:00 | Committing info for 1000 documents/articles
  | 2021-02-03T07:13:30.596-05:00 | Committing info for 1000 documents/articles
  | 2021-02-03T07:13:30.596-05:00 | Committing info for 1000 documents/articles
  | 2021-02-03T07:13:30.596-05:00 | Committing info for 1000 documents/articles
  | 2021-02-03T07:13:30.596-05:00 | Committing info for 1000 documents/articles
  | 2021-02-03T07:13:30.596-05:00 | Committing info for 1000 documents/articles
  | 2021-02-03T07:13:30.596-05:00 | Committing info for 1000 documents/articles
  | 2021-02-03T07:13:30.596-05:00 | Committing info for 1000 documents/articles
  | 2021-02-03T07:13:30.596-05:00 | Committing info for 1000 documents/articles
  | 2021-02-03T07:13:30.596-05:00 | Committing info for 1000 documents/articles
  | 2021-02-03T07:13:30.596-05:00 | Committing info for 1000 documents/articles
  | 2021-02-03T07:13:30.596-05:00 | Committing info for 1000 documents/articles
  | 2021-02-03T07:13:30.596-05:00 | Committing info for 1000 documents/articles
  | 2021-02-03T07:13:30.596-05:00 | Committing info for 1000 documents/articles
  | 2021-02-03T07:13:30.596-05:00 | Committing info for 1000 documents/articles
  | 2021-02-03T07:13:30.596-05:00 | Committing info for 1000 documents/articles
  | 2021-02-03T07:13:30.597-05:00 | Traceback (most recent call last):
  | 2021-02-03T07:13:30.597-05:00 | File "opasDataLoader.py", line 626, in <module>
  | 2021-02-03T07:13:30.597-05:00 | main()
  | 2021-02-03T07:13:30.597-05:00 | File "opasDataLoader.py", line 484, in main
  | 2021-02-03T07:13:30.597-05:00 | ocd.db.commit()
  | 2021-02-03T07:13:30.597-05:00 | AttributeError: 'NoneType' object has no attribute 'commit'

I ran the same forward/backward two process run on my local machine here for PEPArchive (I run PEPCurrent and the other separately).  Here is the log of one of the two processes.  It took 5 hours, and better than that, successfully completed.
TCC LE  14.00.9 x64   Windows 10 [Version 6.3.19041]
Copyright 2016 JP Software Inc.  All Rights Reserved

alias /r c:\win32app\4NT4\aliases.txt

    opasDataLoader - Open Publications-Archive Server (OPAS) - Document, Authors, and References Core Loader

        Load articles into the Docs, Authors, and Glossary Solr cores and
        load article information and bibliography entries into a MySQL database
        (RDS or MySQL) as configured in localsecrets). This data loader is specific to the
        PEP Data and Bibliography schemas but can serve as a model or framework for other schemas.

        See documentation at:
          https://github.com/Psychoanalytic-Electronic-Publishing/OpenPubArchive-Content-Server/wiki/Loading-Data-into-OpenPubArchive

        Example Invocation:
                $ python opasDataLoader.py

        Important option choices:
         -h, --help  List all help options
         -a          Force update of files (otherwise, only updated when the data is newer)
         --sub       Start with this subfolder of the root (can add sublevels to that)
         --key:      Do just one file with the specified PEP locator (e.g., --key AIM.076.0309A)
         --nocheck   Don't prompt whether to proceed after showing setting/option choices

        Example:
          Update all files from the root (default, pep-web-xml) down.  Starting two runs, one running the file list forward, the other backward.
          As long as you don't specify -a, they will skip the ones the other did when they eventually
          cross

             python opasDataLoader.py
             python opasDataLoader.py --reverse

          Update all of PEPCurrent

             python opasDataLoader.py -a --sub _PEPCurrent

        Note:
          S3 is set up with root pep-web-xml (default).  The root must be the bucket name.

          S3 has subfolders _PEPArchive, _PEPCurrent, _PEPFree, _PEPOffsite
            to allow easy processing of one archive type at a time simply using
            the --sub option (or four concurrently for faster processing).

Messaging verbose:  False
Input data Root:  X:\_PEPA1\_PEPa1v
Input data Subfolder:  _PEParchive
Reset Core Data:  False
Forced Rebuild - All files added, regardless of whether they are the same as in Solr.
********************************************************************************
Database will be updated. Location: localhost
Solr Full-Text Core will be updated:  http://localhost:8983/solr/pepwebdocs
Solr Authors Core will be updated:  http://localhost:8983/solr/pepwebauthors
Solr Glossary Core will be updated:  http://localhost:8983/solr/pepwebglossary
********************************************************************************
Locating files for processing at X:\_PEPA1\_PEPa1v\_PEParchive with pattern \((bEXP_ARCH1|bSeriesTOC)\)\.(xml|XML)$. Started at (Tue Feb  2 19:27:58 2021).
--------------------------------------------------------------------------------
Ready to import records from 129966 files at path X:\_PEPA1\_PEPa1v\_PEParchive
Processing started at (Tue Feb  2 19:28:23 2021).
--------------------------------------------------------------------------------
halfway option selected.  Processing approximately one-half of the files that match.
-r option selected.  Running the files found in reverse order.
Load process started (Tue Feb  2 19:28:23 2021).  Examining files.
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Committing info for 1000 documents/articles
Halfway mark reached on file list (64984)...file processing stopped per halfway option
Load process complete (Wed Feb  3 00:29:09 2021).
Performing final commit.
Issue updates.  Writing file ./dataUpdateLogs/updated_issues_20210203-002911.xml
Finished! Imported 64985 documents and 15 references. Total file inspection/load time: 18048.07 secs (300.80 minutes.)
...Files loaded per Min: 216.0397
...Files evaluated per Min (includes skipped files): 432.0661
Note: File load time is not total elapsed time. Total elapsed time is: 18048.07 secs (300.80 minutes.)
Files per elapsed min: 216.0397

@nrshapiro I'm looking into the mysql error you were seeing to try to get you back up and running first and foremost and I believe the issue is that having the loaders run for so long exhausted the RDS instance I/O burst balance. This balance is dependent on volume size and takes time to reaccumulate when used. To start, I've increased the development RDS volume size (you were running low on disk space anyway). Once the burst balance regenerates a bit you should have better database performance.

@lhorne-gavant @adistasio

The processes that you restarted a few hours ago are still running ok...that's good.  Did you use the new --halfway option to keep them from needing to go through all the overlapping files?

Another thing I've been noticing, not sure if this has changed for some reason.  You can't see any logs for the processes until there's some sort of error.  For example, the processes you've started aren't listed in the log groups...the last ones listed are from this morning where there was an error.  Is this a change from previous behavior?  What determines when the output appears in the cloudwatch list?  Does it have to cross some length threshold?

@nrshapiro

I did see the new option and my last run is with "-a --halfway".
That is interesting. I have not made any changes to logging. I do believe it is buffered up to a certain amount for efficiency before sending, but having no logs at all until there is an error is no good. I'll see if i can't get us logging more frequently for future runs.

@nrshapiro I just wanted to give you an update on where the data migration changes are at this morning.

I've decreased logging and made it more user friendly since you occasionally check them out.
Updated to fail fast if any container fails
Updated to run doc folders separately instead of running from the parent directory. Last nights incremental load took 4 hours.
Updated such that the --resetcore flag will now cause _PEPFree to be run with resetcore, followed by normal processing based on any other flags you've sent in, minus resetcore. You should be able to use the "commands" var like usual. Let me know of you see any oddities. I'm still testing these changes, but you should be able to trigger pipelines if you'd like.

I took a look at the cloudwatch log sending delays, but haven't come up with a solution yet. That's on the agenda for today as well as working on the email issues.

Thanks Leyla.  The execution time seems good, but I just tried looking
at all the jobs logs from your runs...see one that runs for PEPFree, but
I don't see any other folders run...and all are overrun by the stat
update logging display (which I can certainly limit).  I had to look at
the files in Raw mode to see it because of that--which in itself is not
a problem.
But can you point me to the right log to see the full build info (not
just the stat)?
Thanks,
Neil
------ Original Message ------
From: "lhorne-gavant"<EMAIL_ADDRESS>To: "Psychoanalytic-Electronic-Publishing/OpenPubArchive-Content-Server"
<EMAIL_ADDRESS>Cc: "Neil Shapiro"<EMAIL_ADDRESS>"Mention"
<EMAIL_ADDRESS>Sent: 2/5/2021 9:54:57 AM
Subject: Re:
[Psychoanalytic-Electronic-Publishing/OpenPubArchive-Content-Server]
DevOps Build Process Optimizations (#84)

@nrshapiro https://github.com/nrshapiro I just wanted to give you an
update on where the data migration changes are at this morning.
I've decreased logging and made it more user friendly since you
occasionally check them out.
Updated to fail fast if any container fails
Updated to run doc folders separately instead of running from the
parent directory. Last nights incremental load took 4 hours.
Updated such that the --resetcore flag will now cause _PEPFree to be
run with resetcore, followed by normal processing based on any other
flags you've sent in, minus resetcore. You should be able to use the
"commands" var like usual. Let me know of you see any oddities. I'm
still testing these changes, but you should be able to trigger
pipelines if you'd like.
I took a look at the cloudwatch log sending delays, but haven't come up
with a solution yet. That's on the agenda for today as well as working
on the email issues.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Psychoanalytic-Electronic-Publishing/OpenPubArchive-Content-Server/issues/84#issuecomment-774081861,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABE5BMURWT63KVXGS764V5TS5QBEDANCNFSM4W77W3UA.

@nrshapiro I ran pipelines a few times for testing yesterday. If you look at pipeline 6050 you should see all folders run. You will have to look at it in raw mode because the log is greater than gitlabs capacity. I did notice the stat updaters logging yesterday. Have you by chance changed how you're logging recently?
-------------------- Loading data --------------------
Starting default loaders
0b54f67936660be6d707a2aa7339291bdb30105d4d4e935736c7947219a612c3
Started container "pep-data-loader-6050-0" with commands "  --sub _PEPFree"
4120f78587e5f3c7a13b795c914f6bd966fab04d1d8c6fc7a04b9e51be90ce74
Started container "pep-data-loader-6050-1" with commands "  --sub _PEPOffsite"
089b0eb97bc1d61bf027539fbfed1c19b39da32089b65f9b9139494039cfc8ad
Started container "pep-data-loader-6050-2" with commands "  --sub _PEPCurrent"
602f4c3dc4cbad092ebb526097acb93fe8289bb0e4a622b1de4873bb5fd046ca
Started container "pep-data-loader-6050-3" with commands "  --sub _PEPArchive --halfway"
66fd7b679c29d13b8dbc87127c24a419eac216bf6b104d52a9c5749ba76427cb
Started container "pep-data-loader-6050-4" with commands "  --sub _PEPArchive --reverse --halfway"

Yes, I changed the way it worked so only records with data are getting
updated.  But that message (error trap for a missing document or stat
field) never gets hit when I run it here.  It should look like the
example below...but I'm going to try running it remotely and see what
happens with the data that's online.
opasDataUpdateStat version 2021.0204.1 started at 2021-02-05 10:38:26.
2021-02-05 10:38:26 opasDataUpdateStat open_connection 126 - DEBUG
Database opened by (get_most_viewed_crosstab) Specs: opascentral for
host localhost,  user root port 3306
2021-02-05 10:38:26 opasDataUpdateStat close_connection 143 - DEBUG
Database closed by (get_most_downloaded_crosstab)
Collecting citation counts from cross-tab in biblio database...this will
take a minute or two...
2021-02-05 10:38:26 opasDataUpdateStat open_connection 126 - DEBUG
Database opened by (collect_citation_counts) Specs: opascentral for host
localhost,  user root port 3306
2021-02-05 10:38:38 opasDataUpdateStat close_connection 143 - DEBUG
Database closed by (collect_citation_counts)
Merging up to 44813 stat records into Solr Docs core records.
...44000 records to go
...43000 records to go
...42000 records to go
...41000 records to go
...40000 records to go
...39000 records to go
...38000 records to go
...37000 records to go
...36000 records to go
...35000 records to go
...34000 records to go
...33000 records to go
...32000 records to go
...31000 records to go
...30000 records to go
...29000 records to go
...28000 records to go
...27000 records to go
...26000 records to go
...25000 records to go
...24000 records to go
...23000 records to go
...22000 records to go
...21000 records to go
...20000 records to go
...19000 records to go
...18000 records to go
...17000 records to go
...16000 records to go
...15000 records to go
...14000 records to go
...13000 records to go
...12000 records to go
...11000 records to go
...10000 records to go
...9000 records to go
...8000 records to go
...7000 records to go
...6000 records to go
...5000 records to go
...4000 records to go
...3000 records to go
...2000 records to go
...1000 records to go
...0 records to go
Finished updating Solr stat with 23 article records updated; records
skipped: 0.
Fri Feb  5 10:44:57 2021 Updated 23 Solr records in<PHONE_NUMBER>071289
secs (6.525685246785482 minutes)).
------ Original Message ------
From: "lhorne-gavant"<EMAIL_ADDRESS>To: "Psychoanalytic-Electronic-Publishing/OpenPubArchive-Content-Server"
<EMAIL_ADDRESS>Cc: "Neil Shapiro"<EMAIL_ADDRESS>"Mention"
<EMAIL_ADDRESS>Sent: 2/5/2021 10:36:37 AM
Subject: Re:
[Psychoanalytic-Electronic-Publishing/OpenPubArchive-Content-Server]
DevOps Build Process Optimizations (#84)

@nrshapiro https://github.com/nrshapiro I ran pipelines a few times
for testing yesterday. If you look at pipeline 6050 you should see all
folders run. You will have to look at it in raw mode because the log is
greater than gitlabs capacity. I did notice the stat updaters logging
yesterday. Have you by chance changed how you're logging recently?
-------------------- Loading data --------------------
Starting default loaders
0b54f67936660be6d707a2aa7339291bdb30105d4d4e935736c7947219a612c3
Started container "pep-data-loader-6050-0" with commands "  --sub _PEPFree"
4120f78587e5f3c7a13b795c914f6bd966fab04d1d8c6fc7a04b9e51be90ce74
Started container "pep-data-loader-6050-1" with commands "  --sub _PEPOffsite"
089b0eb97bc1d61bf027539fbfed1c19b39da32089b65f9b9139494039cfc8ad
Started container "pep-data-loader-6050-2" with commands "  --sub _PEPCurrent"
602f4c3dc4cbad092ebb526097acb93fe8289bb0e4a622b1de4873bb5fd046ca
Started container "pep-data-loader-6050-3" with commands "  --sub _PEPArchive --halfway"
66fd7b679c29d13b8dbc87127c24a419eac216bf6b104d52a9c5749ba76427cb
Started container "pep-data-loader-6050-4" with commands "  --sub _PEPArchive --reverse --halfway"
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Psychoanalytic-Electronic-Publishing/OpenPubArchive-Content-Server/issues/84#issuecomment-774107621,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABE5BMQVG6YVZ6EBBPUR2PTS5QGALANCNFSM4W77W3UA.

From the last run, it looks right.
-------------------- Loading data --------------------
Starting default loaders
5058ee401dd1be5cec26df520ec29181b8c1e683b6d4880083f6022225fe29db
Started container "pep-data-loader-6064-0" with commands "  --sub _PEPFree"
09286bc982a5f4775d6a32528f75f3b9c247225a2bb88130e276273b1806dd01
Started container "pep-data-loader-6064-1" with commands "  --sub _PEPOffsite"
e58583f0288b12b8b5f24dd70977dd7fcada339536bf45dd5edcb6f627a2700c
Started container "pep-data-loader-6064-2" with commands "  --sub _PEPCurrent"
795fe95e8c7977205c04902732dd092ed49c5795321253a5541c9e81073a12be
Started container "pep-data-loader-6064-3" with commands "  --sub _PEPArchive --halfway"
e34dd1c2746bbfe990908ae1f84118da455bc1b3601b65d308d67d2e272f33f4
Started container "pep-data-loader-6064-4" with commands "  --sub _PEPArchive --reverse --halfway"
Waiting for container(s) to complete...
...................................................
Successfully loaded data
-------------------- Restarting Solr --------------------
Stopping old docker container
pep-solr
pep-solr
Removing old docker container
pep-solr
Starting new container
ae1d94e85c841e2587a47fdc35e622ef935441c45d50f88cc3a03ad4e19f5a64
-------------------- Running Stat Updater --------------------
opasDataUpdateStat - Program to update the view and citation stat fields in the pepwebdocs
database.
  By default, it only updates records which have views data.
  If you use command line option --all, it will add all citation and views data to the
    pepwebdocs data.  (This takes significantly longer.)

     - The first update after a load should be with option --all
     - Then, omit --all to update views daily
     - Citations (include --all) only need be updated after data updates

     For complete details, see:
      https://github.com/Psychoanalytic-Electronic-Publishing/OpenPubArchive-Content-Server/wiki/Loading-Data-into-OpenPubArchive
  The records added are controlled by the database views:
     vw_stat_docviews_crosstab
     vw_stat_cited_crosstab

  2020-10-29 Important update - Since it is used at database build/rebuild time, it now updates all records
     where views are non-zero, not just the last week.

opasDataUpdateStat version 2021.0205.1 started at 2021-02-05 20:24:30.
Collecting citation counts from cross-tab in biblio database...this will take a minute or two...

Fri Feb  5 20:29:15 2021 Updated 128 Solr records in<PHONE_NUMBER>3510437 secs (4.742529340585073 minutes)).
-------------------- End Load solr data --------------------
Running after_script
00:02
Running after script...
$ bash utilities.sh CleanupDockerBuild "$SvcName-loader-$CI_PIPELINE_ID"
-------------------- Docker cleanup --------------------
Stopping container(s)
"docker stop" requires at least 1 argument.
See 'docker stop --help'.
Usage:  docker stop [OPTIONS] CONTAINER [CONTAINER...]
Stop one or more running containers
"docker rm" requires at least 1 argument.
See 'docker rm --help'.
Usage:  docker rm [OPTIONS] CONTAINER [CONTAINER...]
Remove one or more containers
Removing image(s)
Untagged: pep-data-loader-6064:latest
Deleted: sha256:7427e4b9e2d2f99b405113a44b2aebd832ce5278cb0622602eaf0bf166fcf584
Deleted: sha256:d942427b21d56966208b7e1cec999f801fab09a3f987017da16115ab13527704
Deleted: sha256:d603ebda2b6b744f730e54c427d79061361b5cfc9f20bad787a9e45e55952d38
Deleted: sha256:76144d5f30b0e35102bdf0aaee64e6153d2153348bb6fad73a983293a059019c
Deleted: sha256:35b855c4336e960bb780844048a94a4d30857dfd01c8903d694d8a3ab0a33996
Deleted: sha256:78f3fcb7a84cb867b2e08164b4f35f142e8d6f9417f71fd9e996b267619d07df
Deleted: sha256:5e0e26c66c2912746cbfd2e89ec91a3192675312451055db13dc631c621d04f5
Deleted: sha256:93be0129ac3f22fb5f36b60e618d64a3ac60ea925ae88cc92b7d01b141d6456a
Deleted: sha256:e037e0d134993ad60275a57c12b4042ba608d5998b523199544f61991f65fffa
Deleted: sha256:118f61bef2489be224549b2dc46904a12e009d34137f6efa8d4458d48e9b4be4
Deleted: sha256:6012991a972a959b24030aa1fee8e3e6b615e78bca1ed8c5f47df24412c37ec2
After cleanup
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
REPOSITORY                          TAG                 IMAGE ID            CREATED             SIZE
pep-data-loader-6030                latest              31730ab774c6        42 hours ago        1.26GB
pep-data-loader-5967                latest              e67a95156d0a        3 days ago          1.26GB
tiangolo/uvicorn-gunicorn-fastapi   python3.7           57c75d7c41d8        8 months ago        970MB
-------------------- End Docker cleanup --------------------
Job succeeded

Not much in the above record about the actual (incremental) data load, so I looked in Cloudwatch for the results, and that looks good.  It only found two new data items, and thankfully, the total time to walk through 130K records to check was just under 40 minutes.  Excerpted here:

  | 2021-02-05T19:01:22.242-05:00 | Skipped 128800 so far...loaded 2 out of 129965 possible.
-- | -- | --
  | 2021-02-05T19:01:22.242-05:00 | Skipped 128900 so far...loaded 2 out of 129965 possible.
  | 2021-02-05T19:01:22.242-05:00 | Skipped 129000 so far...loaded 2 out of 129965 possible.
  | 2021-02-05T19:01:22.242-05:00 | Skipped 129100 so far...loaded 2 out of 129965 possible.
  | 2021-02-05T19:01:22.242-05:00 | Skipped 129200 so far...loaded 2 out of 129965 possible.
  | 2021-02-05T19:01:22.242-05:00 | Skipped 129300 so far...loaded 2 out of 129965 possible.
  | 2021-02-05T19:01:22.242-05:00 | Skipped 129400 so far...loaded 2 out of 129965 possible.
  | 2021-02-05T19:01:22.242-05:00 | Skipped 129500 so far...loaded 2 out of 129965 possible.
  | 2021-02-05T19:01:22.242-05:00 | Skipped 129600 so far...loaded 2 out of 129965 possible.
  | 2021-02-05T19:01:22.242-05:00 | Skipped 129700 so far...loaded 2 out of 129965 possible.
  | 2021-02-05T19:01:22.242-05:00 | Skipped 129800 so far...loaded 2 out of 129965 possible.
  | 2021-02-05T19:01:22.242-05:00 | Skipped 129900 so far...loaded 2 out of 129965 possible.
  | 2021-02-05T19:01:22.242-05:00 | Load process complete (Sat Feb 6 00:01:22 2021).
  | 2021-02-05T19:01:22.242-05:00 | Performing final commit.
  | 2021-02-05T19:01:22.242-05:00 | Finished! Imported 2 documents and 36 references. Total file inspection/load time: 2378.58 secs (39.64 minutes.)
  | 2021-02-05T19:01:22.242-05:00 | ...Files loaded per Min: 0.0505
  | 2021-02-05T19:01:22.242-05:00 | ...Files evaluated per Min (includes skipped files): 3278.3778
  | 2021-02-05T19:01:22.242-05:00 | Note: File load time is not total elapsed time. Total elapsed time is: 2378.58 secs (39.64 minutes.)
  | 2021-02-05T19:01:22.242-05:00 | Files per elapsed min: 0.0505


Graphical issue. Can't access all buttons on the right.

See attachement. I can not access all the buttons on the right. I can not scroll down either. I'm on a MacBook Pro 16" running at full resolution.
I'm on KiCad 7.0.0 and using your latest branch.

I have to find a way to debug Apple stuff somehow as I have no Mac to test on ....

@Bouni I'm on OSX and wanted to take a look at fixing this. How to get kicad plugin manager to use the local cloned repo for easy dev iteration? How do you quickly iterate during dev?

Its actually pretty easy, clone this repo and then:
cd ~/Library/Preferences/kicad/scripting/plugins
git clone https://github.com/chmorgan/kicad-jlcpcb-tools.git

Make sure you have no version of the plugin installed via PCM.
Ideally you create a feature branch before you start making changes to the code:
git switch -c fix-macos-ui
# make your changes
git push -u origin fix-macos-ui

When you're done, make PR here on GitHub.
While making changes, the reload plugins function of KiCAD is very handy. I have a custom Keyboard shortcut for it CTRL+SHIFT+R. This allows you to change stuff and test it on the fly (the plugin must be closed for reloading but better than restarting KiCad every time)

Ahh perfect, this helps a ton and of course on the feature branch.

On Jul 21, 2023, at 1:32 AM, bouni @.***> wrote:
Its actually pretty easy, clone this repo and then:
cd ~/Library/Preferences/kicad/scripting/plugins
git clone https://github.com/chmorgan/kicad-jlcpcb-tools.git
Make sure you have no version of the plugin installed via PCM.
Ideally you create a feature branch before you start making changes to the code:
git switch -c fix-macos-ui
make your changes
git push -u origin fix-macos-ui
When you're done, make PR here on GitHub.
While making changes, the reload plugins function of KiCAD is very handy. I have a custom Keyboard shortcut for it CTRL+SHIFT+R. This allows you to change stuff and test it on the fly (the plugin must be closed for reloading but better than restarting KiCad every time)
—
Reply to this email directly, view it on GitHub https://github.com/Bouni/kicad-jlcpcb-tools/issues/330#issuecomment-1645005346, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJH4AATNUNR6I45OJEZLEDXRIH5XANCNFSM6AAAAAA2MQZOEI.
You are receiving this because you are subscribed to this thread.

Fyi the path ended up being:
/Users/cmorgan/Documents/KiCad/7.0/scripting/plugins
Discovered when trying to figure out why it wasn’t showing up and then clicked on “Reveal plugins folder in finder”.

On Jul 21, 2023, at 3:23 PM, Chris Morgan @.***> wrote:
Ahh perfect, this helps a ton and of course on the feature branch.

On Jul 21, 2023, at 1:32 AM, bouni @.***> wrote:
Its actually pretty easy, clone this repo and then:
cd ~/Library/Preferences/kicad/scripting/plugins
git clone https://github.com/chmorgan/kicad-jlcpcb-tools.git
Make sure you have no version of the plugin installed via PCM.
Ideally you create a feature branch before you start making changes to the code:
git switch -c fix-macos-ui
make your changes
git push -u origin fix-macos-ui
When you're done, make PR here on GitHub.
While making changes, the reload plugins function of KiCAD is very handy. I have a custom Keyboard shortcut for it CTRL+SHIFT+R. This allows you to change stuff and test it on the fly (the plugin must be closed for reloading but better than restarting KiCad every time)
—
Reply to this email directly, view it on GitHub https://github.com/Bouni/kicad-jlcpcb-tools/issues/330#issuecomment-1645005346, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJH4AATNUNR6I45OJEZLEDXRIH5XANCNFSM6AAAAAA2MQZOEI.
You are receiving this because you are subscribed to this thread.

@Bouni alright so I'm able to edit and reload and looking into the issue. It appears that the toolbar tools added with the 2.0 scale factor (the value of self.scale_factor) are simply too large to fit on the screen. They are quite large for me. Maybe 1.5 is a more natural value. In any case, the toolbar apparently doesn't realize them to fit in the available height, and in fact the wxpython docs don't mention toolbars resizing their elements or scrolling them if there isn't enough room.
We could limit the scaling size to 1.5, or not scale at all, or maybe I'm missing something like adding the toolbar to a scrolling container?
Today is my first day with wxpython/wxwidgets so looking for ideas.

I can have a look at it on Monday, don't remember how the scaling works. Just that it was a pain in the rear to get it working 🫤

@Bouni you know, a scale factor of 1 looks perfect here, screenshot attached. Maybe this scaling isn't necessary anymore now that wxwidgets is high dpi aware?

@chmorgan I just checked how we get the scale_factor.
Its all in here: https://github.com/Bouni/kicad-jlcpcb-tools/blob/main/helpers.py#L38-L42
Would you mind checking what's the return value of window.GetDPIScaleFactor() ?
I guess its 2.0 or something like that on OSX

@Bouni it is 2.0 here.
Looking at how things render it doesn't seem correct though to scale icons by that amount.

From this temporary code added to mainwindow.py:
        self.logbox.write(f'{self.scale_factor}')
        self.logbox.write(f'scale factor {self.window.GetDPIScaleFactor()}')

@chmorgan @serpent213 Would one of you guys mind testing #354 and see if that fixes the scaling issue?

@Bouni looks good here for me. Agree that it doesn't seem like osx needs the scaling for whatever reason, maybe because wxwidgets handles it somehow.

Let me know as soon as you have a new compiled version. I could give it a try!

@FantomXR The latest realease should include the fix, let me know if thats the case

@Bouni would recommend this be closed with the changes to the scaling that I had made.


Some issues and implementation details

Thank you very much for sharing your excellent work! I 'd like to record some implementation details.

Some files may be missing in mmdet3D/models.
I've tried to supplement them with the files from the autoalignv2 project.

In mmdet3D/models./init.py, I had to delete line 20 (from .vtransforms import *) to make sure it can successfully run.

However, I still have some questions:

In the project, I can't find the Point-to-Grid Transformer module you've mentioned in your paper.
How can you get the checkpoint of the baseline Transfusion which can achieve 65.1mAP?  Did you train it from scratch?  The checkpoint (IS-Fusion_epoch_10.pth) you provided seems like final weights rather than a pre-trained model.

I'm sorry for taking up your time, and I hope to receive your reply.

Thanks for pointing it out. It seems there was a mismatch between the remote codebase on my server. Hope you’ve fixed it right now.

We have identified that the Point-to-Grid Transformer consumes excessive memory, making it unsuitable for V100 GPUs. Therefore, we have removed it from the released version to ensure smooth operation on commonly used V100 GPUs. It's worth noting that in the original paper, we utilized 8*A100 (80G) GPUs for optimal performance.
The reported performance of 65.1 mAP for Transfusion-L was achieved by removing the image modality. Since IS-Fusion primarily focuses on multimodal implementation, we have excluded this step from the released version. However, you can easily achieve it by removing the image input.
If you encounter any further issues or have additional questions, please don't hesitate to let me know. Thanks.

@yinjunbo
Hi, will the deletion of the Point to Grid Transformer module affect the final performance?
I am trainning the model, worrying that can't meet your proposed performance.


Could not resolve placeholder 'xxxx' in string value "${yyyy}" when used placeholder resolver in spring configurations

java.lang.IllegalArgumentException: Could not resolve placeholder 'xxxx' in string value "${yyyy}" when using configuration placeholder in spring based project (at least in spring xml since I am using spring integration in my project). The exception started to happen when i added springfox-swagger2 dependency to maven project. After digging into the problem i found out that there might be a problem with PropertySourcesPlaceholderConfigurer and set: setIgnoreUnresolvablePlaceholders(true) the error disappeared. The modified code in SwaggerCommonConfiguration.java from springfox-swagger-common for my solution is below:
@Bean
  public static PropertySourcesPlaceholderConfigurer swaggerProperties() {
        PropertySourcesPlaceholderConfigurer p = new PropertySourcesPlaceholderConfigurer();
        p.setIgnoreUnresolvablePlaceholders(true);
        return p;
  }

Tools i used:

java version "1.8.0_31"
Java(TM) SE Runtime Environment (build 1.8.0_31-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.31-b07, mixed mode)
Spring Integration 4.0.6.RELEASE
Spring Context 4.0.6.RELEASE
Spring Boot 1.1.5.RELEASE
Apache Maven 3.0.5
OS Linux Mint 17.1 64-bit
Springfox Swagger 2.0.0-SNAPSHOT

@sabyrzhan good catch! This need to be fixed before 2.0 is released.

@sabyrzhan trying to find out what exact condition causes this - have you any special propertyplaceholder beans or configuration?

So how is it in my project:

I used <context:property-placeholder location="file://${property_location}" /> in my spring-integration XML file.
XML file locate in classpath: /META-INF/integration-context.xml
And imported that file in my configuration bean:

@Configuration
@ImportResource("classpath:/META-INF/integration-context.xml")
public class IntegrationConfiguration {
    private static final Logger LOGGER = LoggerFactory.getLogger(IntegrationConfiguration.class);
.
.
.

@adrianbk It's not clear to me why this a problem. This change will cause property resolution to fail silently. Is that what we want?

@dilipkrish Does not this let other resolvers to do property resolution if $this resolver cannot do that?

@dilipkrish as far as I am aware spring uses all PropertySourcesPlaceholderConfigurer's (PPC) in a cain to resolve a given property. If the springfox PPC happens to be used first to resolve some non-springfox property then, without this change,  an Exception is thrown.

Because the placeholder properties we are using have defaults:"${springfox.documentation.swagger.v1.path:/api-docs}" we don't break other PropertySourcesPlaceholderConfigurer which don't haveignoreUnresolvablePlaceholders set to true.

@adrianbk @sabyrzhan leaving unresolved variables seems wrong to me some how. What do you guys think about scoping the resolution via a prefix and suffix.
something like $springfox{springfox.documentation.swagger.v1.path:/api-docs} that way its scoped and ignoring will not affect downstream libraries? scoping = it will only try to resolve properties that begin with $springfox.

not sure the end result would be much different. is there any way around this which doesn't involve using a PPC?

Unfortunately not :( let me add the scoping prefix. It will at the very least not try and resolve every variable. Springboot has a nice way around it but that won't work with vanilla spring apps

Configure PropertySourcesPlaceholderConfigurer in the container or bean
Refer More for easy configuration
https://in.techiehints.com/wps/2017/12/17/property-file-configuration-for-java-bean-in-spring/

Thanks @sandeepg. This is an old issue however and this problem is solved in 2.7.0

Thanks, how ever i saw more questions discussion in other sites as well
thought of sharing :)
Thanks
Sandeep G
On Dec 17, 2017 10:22 PM, "Dilip Krishnan"<EMAIL_ADDRESS>wrote:

Thanks @sandeepg https://github.com/sandeepg. This is an old issue
however and this problem is solved in 2.7.0
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/springfox/springfox/issues/677#issuecomment-352269193,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHflZz1RosmOjswgtTBBgtUG3F8iHf-nks5tBUavgaJpZM4EA5ki
.


Margin of ambassador page fixed and made responsive

Fixes Issue
https://github.com/kubesimplify/website/issues/166
Screenshots

Note to reviewers
The website is now according to the design and is fully responsive.

The apply now template is repeating i.e. also present in the home page
should we keep that?

I am not sure about it. We can ask @saiyam1814 about it. If he want we can remove it.


Selection is partially broken

In the current master branch you cannot select text by dragging the mouse in the text area.
This is working with the the older version of this plugin: http://mindmup.github.io/bootstrap-wysiwyg/

I'm able to select text, just not able to type anything as every key (no matter what key I press) just turns bold on and off.

@akoskm did you include jQuery.hotkeys in your project? Once I added that everything started working for me.

@akoskm Were you able to get this to work?

@RandomlyKnighted no, in the mean time I think we just switched to another editor. I'll check and report you back.

@RandomlyKnighted we found no reliable text editors so we implemented a plain textarea/preview editor similar to this comment box.


FinC.ignore_unused() doesn't work for ignoring the "self" parameter

I have a Fin class with some functions that do not use class members. In the generated code I got compiler warnings because the "self" parameter wasn't used in the generated code.
// Fin implementation
private void helper(u32 var1, u32 var2)

// Generated code
void PRIVATE_sample_Class_helper(sample_Class* self, uint32_t var1, uint32_t var2) <-- gcc produces a warning due to self not being used.

To ignore the warning, I added this line to my class:
FinC.ignore_unused(this);

But unfortunately, it generated this code, which does not compile because the parameter is called “self” in the generated code:
(void)(this);

I wanted it to generate this code:
(void)(self);

To workaround this issue, I have used the FinC.EchoToC() function
FinC.EchoToC("  (void)self"); // Ignore "unused variable" compiler warning in generated code.

thanks! Fixed in a2746b4.

This worked for me! Just a thought, maybe Fin could automatically add that line in the future if it sees that the function doesn't use "this" in the body?


ESP-IDF bootloader format

ESP-IDF bootloader format is essentially a binary format with a header and splits the application into different segments. Read more here.
Overall I feel like this has gone in quite cleanly, and seems to work well from my limited testing (minus the fact that the current esp32c3 bootloader is slow as hell).
I don't like having to depend on xmas elf. We should make this internal to espflash or use the object crate in espflash.
Whilst adding support for the esp-idf bootloader, I also refactored the cli module a bit.

cargo run -p probe-rs-cli -- run --chip esp32c6 --format idf /path/to/elf

@MabezDev, is there any signature/marker we could look for that could help us identify programatically when this is an IDF format? I'm thinking it would be nice if the user doesn't need special config settings ... it just makes downstream support and documentation a little easier :)

I would prefer to merge an initial implementation, and work on auto-detection separately.

@Tiwalun @Yatekii rebased on master, I think this should be good to go :)

Sorry mate, I messed it up again with another merge :(

No worries! Rebased again :)


Return hlint code actions as type 'quickfix'

Closes #1524

The Appveyor failure is in their environment, all other CI passes, I am going to merge. It is a trivial change, I did test it against vscode locally.


⚠ 한경주요지수-코스피200 has degraded performance

In 4e1f362, 한경주요지수-코스피200 (https://markets.hankyung.com/indices/kospi200) experienced degraded performance:

HTTP code: 200
Response time: 7414 ms

Resolved: 한경주요지수-코스피200 performance has improved in e089a69.


Show all features in documentation using docsrs

fixes #64
This will show all Modules, Traits, ... that are behind feature flags and add a  "Available on crate feature host only." label on https://docs.rs.
Since cargo doc is also run in the CI, we might want to update this to a (nightly) version that also supports the --cfg docsrs flag, e.g.
      - uses: dtolnay/rust-toolchain@nightly
      - run: cargo doc --workspace --all-features --no-deps --document-private-items
        env:
          RUSTDOCFLAGS: "--cfg docsrs -D warnings"

Need to update cargo doc in CI to use nightly

I've added a docs.rs job which installs the nightly toolchain and does not install cargo-risczero or Foundry.
https://github.com/risc0/risc0-ethereum/pull/78/commits/77a8c66865ef8e295133e0206863c2e53087acb4


"Find average fight" doesn't work

Describe the bug
Clicking the "Find average fight" button spins forever without actually finding anything.
Expected behavior
I would expect the "Find average fight" button to find and display an average fight.
Browser
Bug reproduced with both Chromium 108.0.5359.124 (openSUSE Build) stable (64 bits) and Firefox 102.6.0esr (64 bits) on openSUSE Leap 15.3.
Profile
Bug reproduced with the default "Phase 1 Arcane Alliance" quick profile as well as my own gear+config, so I don't think the profile is relevant for this bug.

This can happen if you click "Find average fight" after changing profile.
Can you reproduce this by clicking the button after finishing a run without changing anything? If so, can you open the web console (F12) and see if there are any error messages?

Thanks for the quick reply. I did not change the profile, clicked straight on "Find average fight" after a 3000-iteration run. This is not a new issue for me by the way, I noticed it several weeks ago, just never bothered reporting until today.
No error message in the Console tab. The Network tab shows a never-ending loop between 3 URL:

OK, I made some progress. I cleared the cached data of my browser, and the bug went away. So I concluded that my personal settings are triggering the bug. After some trial and error, I figured out that this is caused by setting a fixed RNG seed. With the default setting (0), the bug does not happen. If I set it to my favorite number: 42 then the bug happens.
Hopefully you will be able to reproduce the bug by setting your RNG seed to 42.

Yeah find avg doesn't quite make sense with a fixed rng seed. I'm just gonna disable that button when that is set.

Hmm, why is that? As I understand it, the RNG seed is only set for the first run of the series, you still get a different fight with every iteration, with a DPS which could be very different, as shown by the HISTOGRAM tab. So I would expect "Find average fight" to be still useful even with a fixed RNG seed.
On the other hand, I don't know what logic is used to determine what an average fight would be. I naively expected it to be the fight with the DPS as close as possible to the average DPS. Is it checking more than this?


feat: Implement dotWithBraReal

Before submitting
Please complete the following checklist when submitting a PR:

[x] All new features must include a unit test.
If you've fixed a bug or added code that should be tested, add a test to the
tests directory!

[x] All new functions and code must be clearly commented and documented.
If you do make documentation changes, make sure that the docs build and
render correctly by running make docs.

[ ] Ensure that the test suite passes, by running make test.

[x] Add a new entry to the .github/CHANGELOG.md file, summarizing the
change, and including a link back to the PR.

[x] Ensure that code is properly formatted by running make format.

When all the above are checked, delete everything above the dashed
line and fill in the pull request template.

Context:
This functionality is needed to compute adjoint Jacobian of several loss functions that can't be expressed as an observable, e.g. L2 loss and KL divergence. The derivative of those loss functions would require doing adjoint Jacobian on a statevector.
But it should also speed up np.real(np.vdot(bra, ket_temp))
    if op.num_params != 0:
        dU = qml.operation.operation_derivative(op)
        ket_temp = apply_operation(qml.QubitUnitary(dU, op.wires), ket)

        dM = 2 * np.real(np.vdot(bra, ket_temp))
        grads.append(dM)

in https://pennylane.ai/qml/demos/tutorial_adjoint_diff/, so that the dot product happens within GPU without the need to copy to CPU RAM for a NumPy operation.
I optimized this operation because I found it to take a significant time when I did a line profiling on my adjoint differentiation code.
Note: I didn't run make test because it needs a CUQUANTUM_SDK env var which I don't know what to specify (it's not defined even within the cuQuantum Appliance). But at least the output of the adjoint differentiation is correct when I compared with the standard method.
Description of the Change:
This PR implements a dot product of the statevector with another statevector. For the Python binding, I restrict it to real because it seems simpler. Probably should be changed, but I got this error if I returned the whole complex number instead of the real part
TypeError: Unregistered type : float2

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "adjoint_stuff/pennylane_adjoint_dedup.py", line 261, in <module>
    gradient = grad_fn(params)
  File "adjoint_stuff/pennylane_adjoint_dedup.py", line 218, in adjoint_kl
    dM = -2 * bra_norm * dev_gpu._gpu_state.dotWithBraReal(dev_bra._gpu_state)
TypeError: Unable to convert function return value to a Python type! The signature was
        (self: pennylane_lightning_gpu.lightning_gpu_qubit_ops.LightningGPU_C64, arg0: pennylane_lightning_gpu.lightning_gpu_
qubit_ops.LightningGPU_C64) -> float2

Benefits:
Will result in faster adjoint differentiation calculation.
This is the circuit I used to benchmark the L2 loss function adjoint derivative
@qml.qnode(dev, diff_method="finite-diff")
def circuit(params):
    # define your quantum circuit here
    _ = [qml.RX(params[i], wires=i) for i in range(num_wires)]
    _ += [qml.CZ(wires=(i, i + 1)) for i in range(num_wires - 1)]
    _ += [qml.RX(params[i + num_wires], wires=i) for i in range(num_wires)]
    return qml.state()

For 22 qubits, it is 16.7x faster. The first and last numbers are for consistency check that the results match.
first last -0.00015151694036875393 -0.00020264025022278406
Elapsed apply_operation 4.2274839878082275
first last -0.0001515170088100138 -0.00020264028214586442
Elapsed lightning 0.25325512886047363

For 24 qubits, it is ~108x faster
first last -5.618167868933744e-05 -3.4206640947492466e-05
Elapsed apply_operation 97.06205105781555
first last -5.61816643352604e-05 -3.420663656064312e-05
Elapsed lightning 0.8937075138092041

Possible Drawbacks:
N/A
Related GitHub Issues:
N/A
cc; @hthayko

This is the first part of the adjoint differentiation implementation for statevector for the loss functions. Another PR is coming for https://github.com/PennyLaneAI/pennylane.

Hi @rht! Thanks for opening this pull request :star_struck:
Over the past few months, we've had a dedicated effort to combine our lightning ecosystem into a single monorepo: https://github.com/PennyLaneAI/pennylane-lightning
We'll shortly be archiving this repository and redirecting contributions to our monorepo. Please could you open a pull request to the monorepo with these changes to Lightning GPU?
As discussed with @hthayko, we'd be happy to hop on a call to explain any changes needed.

SGTM, I will open the PR on https://github.com/PennyLaneAI/pennylane-lightning.

I'm curious to hear about the changes needed to core PennyLane? Generally we've tried to design PennyLane so that the device can provide custom gradients, such as the adjoint, without having to change the core pipeline.

I wasn't aware that device-specific implementation should be implemented in device-specific repo. But I need to define 2 new classes L2Loss and KLDivergence in https://github.com/PennyLaneAI/pennylane/blob/master/pennylane/qnn/cost.py. And the adjoint code speedup is going to be triggered by checking if isinstance(obj, L2Loss).

Hi @trbromley - thanks for the prompt response.
To give a little bit of context and clarify the big picture, here is what we want to achieve eventually:
dev = qml.device('lightning.gpu', wires=n)
@qml.qnode(dev, diff_method='adjoint')
def adjoint_test(x):
    qml.PauliX(0)
    # ... some gates
    return qml.loss(target_probs, loss_type="L2")

adjoint_test(x) # returns the L2 loss
qml.grad(adjoin_test)(x) # returns the gradient of L2 loss w.r.t. x

Let me know what you think about it.

Current syntax we have working
import pennylane as qml
from pennylane_lightning.lightning_gpu.lightning_gpu import loss

def grad_fn(params):
    with qml.tape.QuantumTape() as tape:
        _ = [qml.RX(params[i], wires=i) for i in range(num_wires)]
        _ += [qml.CZ(wires=(i, i + 1)) for i in range(num_wires - 1)]
        _ += [qml.RX(params[i + num_wires], wires=i) for i in range(num_wires)]
        loss(target_probs, loss_type="L2)
    out = dev_gpu.adjoint_jacobian(tape)
    return out

tic = time.time()
g = grad_fn(params)
toc = time.time()
print("first last", g[0], g[-1])
print("Elapsed", toc - tic)

This is the current implementation we have: https://github.com/PennyLaneAI/pennylane-lightning/pull/562.


use of tilde ~ in Razor scripts

Hi there,
Could you please share any reference to theory or direct explanation of the use of tilde in scripts similar to what Serene has in AccountResetPassword.cshtml.
I do know that when used with path it means relative path reference.
But what does it mean when it is used in id=
Like below - what other object will take button id="~_SubmitButton" or form id="~_Form" action="" and translate it into something else?
Many thanks!!

Every widget has some unique prefix. ~_ is replaced with that prefix. This helps prevent clashes when multiple dialogs with similar ID values are opened, cascaded etc.

This prefix changes everytime, so you should use this.byId("xyz") to find elements, not $('#xyz') or $('#Serene_MyDialog17_xyz") etc


Custom branch for update flake action

No objection, but can someone remind me what this accomplishes?

Branches named update/* are special-cased so that CI only runs for the PR (and not for the branch). Just a small thing.


Add Stremio discord addon

This addon will display the current movie or series as 'game activity' on Discord. It requires the discord-rpc package to be whitelisted though.

I love the idea of this add-on, and even though discord-rpc would probably only ever be used for this one add-on, I'd still add it.
But there are a few things that we need to think about first.

The addonsList.json now auto-updates to all users straight from the raw data of this github repo, as this add-on needs a new module added, we need to state in addonsList.json that this add-on will only work from the next version of PMS, so please set "requires": "1.1.9" for ur add-on in addonsList.json to solve this

discord-rpc has only 2 dependencies: node-fetch and ws. We can even add node-fetch as a dep to this repo too and whitelist it as an alternative http fetcher. There might be a issue with ws though.. Afaik, ws's server implementation uses a binary (unless something changed), and discord-rpc seems to install the entire thing, not just the client. This could or could not be an issue, i'm pretty sure they just need the client for this, so as long as the server isn't used, it shouldn't cry about it's missing binary.. (keep in mind, all binaries are lost when packaging the app, need to be added separately as assets, and i would generally avoid binary dependencies for better cross-platform support)

So, with discord-rpc, the only way i can make sure it will work when the app is packaged, is by packaging and testing the add-on before releasing it. This only means it might take a bit longer before I can be sure there won't be issues so I can merge the PR.
Please set the "requires": "1.1.9" for ur add-on in the meanwhile.

uhm, i see this also uses a module called imdb:
https://github.com/sleeyax/stremio-discord/blob/master/package.json#L12
that is not in the whitelist, can u please change that code to use cinemeta's endpoint?
cinemeta is an internal module though, and i saw u want this add-on to work standalone too, so instead of using the internal module for it, you can just replicate the logic from this file:
https://github.com/sungshon/PimpMyStremio/blob/master/src/lib/cinemeta.js
It should be as easy as changing this line:
https://github.com/sleeyax/stremio-discord/blob/master/addon.js#L21
to: const info = await imdb(args);
and the request in imdb-promise.js should look something like:
needle.get('https://v3-cinemeta.strem.io/meta/' + args.type + '/' + args.id + '.json', (err, resp, body) => {
	if (body && body.meta) {
		resolve(body.meta)
	} else {
		reject(err || Error('Unknown error in cinemeta for request: ' + JSON.stringify(args)))
	}
})

it shouldn't be a hard change, and will help us not whitelist the imdb module for something we already have an internal solution for.

I've created a PR that uses cinemeta:
https://github.com/sleeyax/stremio-discord/pull/1

If u merge that PR and make a new release on github, I'll merge the PR for PMS too.
I've had time to test it with the built app and it works well.

Good to hear that! I'll publish a release right away. Thanks again.

merged! thank you for this fun add-on, I don't think anyone ever thought of using the subtitle handler like this before :))
i think this could of been a great add-on for the add-on contest, it's a fun and original idea

I've published a new release that includes your add-on.


Exporting Visualised Graphs for Further Manipulation

Hi,
Is there a way to export a visualized Netron graph to a text / JSON (or another kind of) file? This might be a great cross-platform way to represent and modify NNs!
Cheers

Can you be more specific or submit a pull request for what you are trying to accomplish? The app supports dozens of formats for representing graphs. Formats like ONNX are intended for model exchange. For visuals there is export to SVG and PNG.


Update index.md

link to starter app was pointing to old repo
What I did

changed link in body text from https://github.com/open-wc/open-wc-starter-app to https://github.com/open-wc/open-wc

While the starter-app itself is a bit out of date, I'm not sure the updated URL you've chosen here is correct, particularly in that it's not an "app" but the entire OWC project... not sure there's a "right" place to link this as we've gone all in on leveraging the generator as opposed to cloning a project.

I agree that this new link it to the whole project, but my logic was that the old link was pointing to the wrong place and the code there hadn't been changed/updated for something like 3 years - and the instructions that I was linking from still apply to the new link, i.e. how to use the generator.