Odd cycles between phi nodes

Binary Ninja Version: 2.4.2875-dev, adb7c635
Platform: macOS 10.15
When analyzing several non-trivial binaries in MLIL SSA, I noticed an oddly looking cycles between phi nodes.
Here is an example:

We can see two types of cycles:

var#1 = phi(var#2, var#1, var#3) - "self" cycle
var#3 = phi(var#2, var#4) <-> var#4 = phi(var#3, var#4, var#5) - "between phi nodes" cycle

Is there a rational explanation for this situation, or is it a bug? I could not explain this from looking at the binary in LLIL SSA nor disassembly.

Unless I misunderstand the question, this is exactly one of the expected features of phi nodes. Phi nodes “merge” data flows together. A dataflow cycle created by phi nodes suggests that a variable is being altered in a loop. Your basic for-loop counter, for example:
for (i=0;i < 4;i++);

i#1 = 0
loop:
i#2 = phi(i#1, i#3)
i#3 = i#2 + 1
if (i#3 < 4) goto loop

Here, the phi node is there to let you know that the value of i#2 is either i#1 or i#3, and that creates a cycles because, well, there is a cycle in that flow.

Unless I misunderstand the question, this is exactly one of the expected features of phi nodes. Phi nodes “merge” data flows together. A dataflow cycle created by phi nodes suggests that a variable is being altered in a loop. Your basic for-loop counter, for example:
for (i=0;i < 4;i++);

i#1 = 0
loop:
i#2 = phi(i#1, i#3)
i#3 = i#2 + 1
if (i#3 < 4) goto loop

Here, the phi node is there to let you know that the value of i#2 is either i#1 or i#3, and that creates a cycle because, well, there is a cycle in that flow.

Yes, the above example is a loop cycle and I understand it. There is no odd phi cycle in it.
But, in the example I provided you can see there is just one SSA def of arg9#4 and at the same time it contains a use of the very same variable and version. Kinda chicken-egg problem.
How should I interpret this?
In the other example I provided there are two phi nodes which constitute variable definitions and both of them also use the defined variables as input. Thus, neither the variable def nor use, appear anywhere else but in the phi node cycle.

Are there any edges back to the first basic block in the function? I'm curious about how the instruction at index 0 is a phi statement in the first place unless there's a back edge there.
Otherwise they look valid, even the instruction with index 52. If you consider a block that ends in a jumptable dispatch that potentially comes back to itself, you could have a phi providing its own input.

Are there any edges back to the first basic block in the function? I'm curious about how the instruction at index 0 is a phi statement in the first place unless there's a back edge there.
Otherwise they look valid, even the instruction with index 52. If you consider a block that ends in a jumptable dispatch that potentially comes back to itself, you could have a phi providing its own input.

The first basic block was the beginning of a function without any back edges.

If the function starts with phi nodes despite there not being other edges to the first basic block in the function that would be a pretty severe bug that I’d be very interested in repros for.
The only situations I’ve ever seen graphs like the one you posted with instruction 0 being a phi node is where a function is tailcall recursive where we’ll wind up just turning the jmp(function_start) into a goto(function_start) which causes the edge back into the entry block (thinking about it now, it probably makes more sense to unconditionally translate those into tailcalls to prevent them from ever contributing edges).
For the rest of it though, those cycles can be expected where loops are present.

Closing as without a repro as near as I can tell this is working with intended, as phi cycles can be expected. As described there could be a bug relating to the phi instructions at the function entry point but I need more information to track that down.


fix creation of first dashboard

first dashboard creation was failing due to no
dashboards being returned from the
"dashboardToClone" function call.
this only affected users with no other existing dashboards.
fixes #474

:+1:


Add lightweight connectivity check on connect()

This PR updates connect() to include an active, lightweight connectivity check for the various services used.
Before, connect() would only create services, to be used when necessary in later calls. A connectivity issue would have only been seen later on, when a sub-service would be used.
An exception was the Greenlight service, which made an active connection during connect for register and recover. In those cases, any connectivity issues to the scheduler would have thrown an error in our connect. However, if the node was already registered and credentials already known, no connection to the Greenlight Scheduler was made during our connect. With this PR, the Greenlight connectivity is checked regardless of the node init method.

Some (all?) of these checks could be outsourced to Breez Server via a global status API (https://github.com/breez/breez-sdk/issues/433#issuecomment-1709891477).
Maybe that's a good compromise to consider:

pros: it reduces the connection checks to one connection and therefore speed up connect
cons: it checks connectivity from the Breez Server to these endpoints, not from the caller

The cons would be acceptable IMO, especially for mobile devices with slow or unreliable data network.


empty string should be a valid value for a virtual document

have a document provider that return ""
💥 an error happens, the empty string should be a valid result

The registerTextDocumentContentProvider API is new in the may release of vscode, right?
So my generated extension can not yet use this API until we release our may version?
Is there some other way to verify this - except writing unit tests which I see you already did?

The registerTextDocumentContentProvider API is new in the may release of vscode, right?

No, we have that since January.


Add stub import doc

Hi, I have created a doc for #1497 .
If you have time, could you please take a look at this PR. Thank you.

Looks good to me.


Handle mounts properly

Fixed by 13be962569d5eaff502dcb257217ef577c98865b

However, we want https://github.com/containerd/containerd/pull/8789 to be merged into containerd.


Create a draft for the docker container structure

Priority: High
Description:
Create a draft for the structre of the wanted docker container structure as researched beforehand.
Estimated time:
2h


ERROR: ld.so: object '/home/vitaly/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.

Hi, full log:
(process:13110): GLib-GIO-WARNING **: 20:09:13.161: Can't find module 'dconf' specified in GSETTINGS_BACKEND
XRRGetOutputInfo Workaround: initialized with override: 0 real: 0xe7aa1dc0
XRRGetCrtcInfo Workaround: initialized with override: 0 real: 0xe7aa0500
CAppInfoCacheReadFromDiskThread took 10 milliseconds to initialize
Steam Runtime Launch Service: starting steam-runtime-launcher-service
Steam Runtime Launch Service: steam-runtime-launcher-service is running pid 13291
bus_name=com.steampowered.PressureVessel.LaunchAlongsideSteam
BRefreshApplicationsInLibrary 1: 0ms
BuildCompleteAppOverviewChange: 129 apps
RegisterForAppOverview 1: 3ms
RegisterForAppOverview 2: 3ms
chdir "/home/vitaly/.local/share/Steam/steamapps/common/Rexaura"
ERROR: ld.so: object '/home/vitaly/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
ERROR: ld.so: object '/home/vitaly/.local/share/Steam/ubuntu12_64/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS64): ignored.
ERROR: ld.so: object '/home/vitaly/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
ERROR: ld.so: object '/home/vitaly/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
pid 13754 != 13753, skipping destruction (fork without exec?)
ERROR: ld.so: object '/home/vitaly/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
ERROR: ld.so: object '/home/vitaly/.local/share/Steam/ubuntu12_64/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS64): ignored.
Adding process 13752 for gameID 317790
src/tcmalloc.cc:289] Attempt to free invalid pointer 0x919fd00
/home/vitaly/.local/share/Steam/steamapps/common/Rexaura/hl2.sh: line 67: 13756 Aborted                 ${GAME_DEBUGGER} "${GAMEROOT}"/${GAMEEXE} "$@"
Uploaded AppInterfaceStats to Steam
Removing process 13752 for gameID 317790
Removing HIDAPI device 'DragonRise Inc. Generic USB Joystick' VID 0x0079, PID 0x0006, bluetooth 0, version 263, serial NONE, interface 0, interface_class 0, interface_subclass 0, interface_protocol 0, usage page 0x0001, usage 0x0004, path = /dev/hidraw5, driver = NONE (DISABLED)
[2024-08-03 20:10:36] Shutdown

The game is Rexaura - the Portal addon.
Gentoo Linux no-multilib, GPU AMD 6800 XT.
It works with https://github.com/ivan-hc/Steam-appimage

src/tcmalloc.cc:289] Attempt to free invalid pointer 0x919fd00

Perhaps this is due to some of the packages that are present in Conty and not present in Steam-appimage, or the fact that glibc is different. I don't know exactly.

i can't reproduce this on my system

Maybe because you have 32 libs.

Those LD_PRELOAD errors happen when you try to use a 32 bit library on a 64 bit binary and viceversa. However it usually doesn't cause issues as it is ignored.
Glibc being different shouldn't be causing the issue either, neither not having 32 bit libs in the host since the container has all of them.
However it is possible that the conty container that's released here has an extra package that's causing issues that is not in the steam appimage.


Inference updates, new features, and bug fixes

Added MCMC diagnostics that get printed out after every run (can be stopped by setting convergence_diagnostics = False). Includes autocorrelation time and acceptance fraction of the chain. Fix for #91 (thanks @mkratz for suggestion)
Initial seed for MCMC chain can now be directly set by the user. Simply passing in the numpy array of initial parameter values to the init_seed argument works. Alternatively, a float number can be passed into this argument that creates a Gaussian ball (of radius passed in as init_seed) around the initial values set in the Model object. Fix for #92
LMFit added as a possible inference method. Detailed documentation in the corresponding notebook in inference examples directory.
Refactored the old "emcee_interface.py" to a new name file "inference_setup.py" since this file sets up inference for emcee and lmfit as well.
Fixed bugs with data for multiple measurements not loading correctly and initial condition related bugs as well (thanks to Cindy and Noah for pointing it out)
For uniform and log uniform priors, I now return the uniform probability density (1/(b-a)) when the values lie within the bounds. This would not change any result as the probability returned is constant, so this change is only for being theoretically correct.
Fixed bug with Gaussian prior in issue #79  (thanks to James Parkin @boospooky for pointing it out)
Added multiple tutorial notebooks for inference, in particular, for least squares fit using LMfit.
Delay support now available with inference to fix #104. (needs a fix for delay simulator). See #103

There is a bug in bioscrape inference getting started:
This cell:
`# from bioscrape.pid_interfaces import
from bioscrape.inference import py_inference
Import data from CSV
Import a CSV file for each experiment run
prior = {'m' : ['gaussian', m_true, 500],'b' : ['gaussian', b_true, 1000]}
sampler, pid = py_inference(Model = M, exp_data = exp_data, measurements = ['y'],
time_column = ['x'], params_to_estimate = ['m','b'],
nwalkers = 32, nsteps = 2000, init_seed = 1e-4, prior = prior,
sim_type = 'deterministic', plot_show = False)
Recommended to simply use sampler object/mcmc_results.csv and generate your own custom plots
truth_list, uncertainty_list = pid.plot_mcmc_results(sampler);`
Produced this error:
`---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
bioscrape\types.pyx in bioscrape.types.Model.check_parameters()
ValueError: Unspecified Parameters: _b=nan, _m=nan
Exception ignored in: 'bioscrape.types.Model._initialize'
Traceback (most recent call last):
File "bioscrape\types.pyx", line 1906, in bioscrape.types.Model.check_parameters
ValueError: Unspecified Parameters: _b=nan, _m=nan
RuntimeError                              Traceback (most recent call last)
RuntimeError: Model has been changed since CSimInterface instantiation. CSimInterface no longer valid.
Exception ignored in: 'bioscrape.simulator.ModelCSimInterface.check_interface'
RuntimeError: Model has been changed since CSimInterface instantiation. CSimInterface no longer valid.
TypeError                                 Traceback (most recent call last)
 in
8                      time_column = ['x'], params_to_estimate = ['m','b'],
9                       nwalkers = 32, nsteps = 2000, init_seed = 1e-4, prior = prior,
---> 10                       sim_type = 'deterministic', plot_show = False)
11
12 # Recommended to simply use sampler object/mcmc_results.csv and generate your own custom plots
bioscrape\inference.pyx in bioscrape.inference.py_inference()
C:\ProgramData\Anaconda3\lib\site-packages\bioscrape\inference_setup.py in run_mcmc(self, **kwargs)
235     def run_mcmc(self, **kwargs):
236         self.prepare_inference(**kwargs)
--> 237         sampler = self.run_emcee(**kwargs)
238         return sampler
239
C:\ProgramData\Anaconda3\lib\site-packages\bioscrape\inference_setup.py in run_emcee(self, **kwargs)
386         assert p0.shape == (self.nwalkers, ndim)
387         sampler = emcee.EnsembleSampler(self.nwalkers, ndim, self.cost_function)
--> 388         sampler.run_mcmc(p0, self.nsteps, progress = progress)
389         if convergence_check:
390             self.autocorrelation_time = sampler.get_autocorr_time()
C:\ProgramData\Anaconda3\lib\site-packages\emcee\sampler.py in run_mcmc(self, pos0, N, rstate0, lnprob0, **kwargs)
170
171         for results in self.sample(pos0, lnprob0, rstate0, iterations=N,
--> 172                                    **kwargs):
173             pass
174
TypeError: sample() got an unexpected keyword argument 'progress'`
Note: I am on the Inference-dev branch via the PR you made.

There is a bug in bioscrape inference getting started:

Looks like you have older emcee version. Can you try updating it and run again?

I added emcee version requirement to requirements.py.

Added ~10 new tests for inference.


Require explicitly defining wikidataQueryService in Kartotherian config

Rather than defaulting to a hardcoded WDQS endpoint in the application
code, we should require explicitly defining it in the service
configuration.  This will help service operators easily identify the
endpoint being used, and help developers keep it up to date.

For context, see T205607: Kartotherian should use discovery endpoint to connect to wikidata query service and other tasks linked there.


Models are getting blurred when zoomed out.

I have added a low poly model with a 512px texture and its edges are getting blurred?

Please Help ?

Can you please help me in resolving this?

The quality of the rendered model depends on the capabilities of your pc, browser, video card but also on the model itself. This has barely anything to do with blueprint3d itself, you can close this issue.

I have already fixed this issue a long back, sorry for not updating the issue. Here I am talking about blurring not quality, blurring happens when the device pixel ratio is not set. So have to add one line in three/main.ts i.e renderer.devicePixelRatio = window.devicePixelRatio


[Refactor][admin] deprecate namespace

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description
No response
Use case
No response
Related issues
No response
Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

I want to work on this issue. Please assign me this issue.


Difference between rotor($batch, :partial) and batch isn't described enough

The problem
See:

https://docs.perl6.org/routine/rotor
https://docs.perl6.org/routine/batch

The batch document says:
If the number of elements is not a multiple of $batch, the last batch may have less than $batch elements, similar to .rotor($batch, :partial).

This explanation just say they are "similar" and doesn't provide a hint which one to use (i.e., batch or rotor($batch, :partial).
Suggestions

If there is a difference: This explanation should provide a hint which one to use.
If there aren't any differences: This explanation should say they are same.

Thanks for the report. We'll try to address it ASAP.


Evaluation forms list

User story
As a challenge manager, in order to manage all my evaluation forms efficiently, I would like to be able to review them in the list view and understand what the forms are for, and what actions I can take.
Acceptance Criteria:

[x] The challenge manager can navigate to a list of evaluation forms via the dashboard or utility nav
[x] If the user does not have any saved evaluations forms, the list is empty

[x] The user can start the process for creating a new evaluation form from this page

[x] If the user has saved evaluations forms, the list of evaluation forms displays:

[x] Evaluation form name
[x] Challenge name the form is assigned to (if any) (note: challenge name is not a link to the challenge in Phoenix)
[ ] Evaluation period dates:

[ ] Start date is the end date of the challenge the evaluation form is assigned to
[ ] End date is set by the challenge manager during form creation

[x] The user can choose the following options:

[x] Edit the form (the form cannot be edited (expect evaluation end date) if it's within or after evaluation period), #61
[x] ~Delete the form (the form cannot be deleted if it's within or after evaluation period), #63~
[ ] Post-MVP: Clone, #62
[ ] Post-MVP: Preview

[x] From the list view, the user can select to create a new evaluation form, #61

Development tasks:

[x] #129
[x] #130
[x] #236

Definition of Done
Doing (dev team)

[x] Code complete
[x] Code is organized appropriately
[x] Any known trade offs are documented in the associated GH issue
[x] Code is documented, modules, shared functions, etc.
[x] Automated testing has been added or updated in response to changes in this PR
[x] The feature is smoke tested to confirm it meets requirements
[x] Database changes have been peer reviewed for index changes and performance bottlenecks
[ ] PR that changes or adds UI

[ ] include a screenshot of the WAVE report for the altered pages
[x] Confirm changes were validated for mobile responsiveness

[x] PR approved / Peer reviewed
[x] Security scans passed
[x] Automate accessibility tests passed
[x] Build process and deployment is automated and repeatable
[x] Feature toggles if appropriate
[x] Deploy to staging
[x] Move card to testing column in the board

Staging

[ ] Usability testing: mobile and desktop (Tracy or Marni)
[ ] Accessibility tested (Marni)

[ ]  Keyboard navigation
[ ]  Focus confirmed
[ ] Color contrast compliance
[ ] Screen reader testing

[ ] Cross browser testing - UI rendering is performant on below listed devices/browsers (Tracy or Marni)

[ ] Windows/Chrome
[ ] Windows/Edge
[ ] Mac/Chrome
[ ] Mac/Safari
[ ] iOS/Safari

[ ] AC review (Renata)
[ ] Deploy to production (production-like environment for eval capability) (dev team)
[ ] Move to production column in the board

Production

[ ] User and security documentation has been reviewed for necessary updates (Renata/Tracy/Dev team)
[ ] PO / PM approved (Jarah or Renata)
[ ] AC is met and it works as expected (Jarah or Renata)
[ ] Move to done column in the board (Jarah or Renata)

Figma link (see CM Evaluation): https://www.figma.com/design/VfdBFG8ShVnOStdSNYy9fR/Challenge.Gov-Component-Library?node-id=909-27788&t=bN3FSfpATePzF43T-0

How does the user navigate here? Can we add that to the ticket?

How does the user navigate here? Can we add that to the ticket?

@stepchud The user navigates to here from here: https://github.com/GSA/Challenge_platform/issues/53

I think there will be a lot of chicken/egg type problems like this, but I'm wondering purely from execution perspective how can we plan for delivering this in stages, i.e. without having the full functionality of the page and all the form cards implemented.

@stepchud Let's discuss this. I broken down the user stories so they can be delivered in stages, but if that does not address your concern I would like to learn more.

@stonefilipczak Do you have any recommendations on how to test this, given that the form creation work is not completed yet, and I cannot really create a form to have a list of forms. Should we wait until #61 and #65 are completed to test this? If so, that is is absolutely acceptable, but I was wondering if you know of an alternative way to test it.

Issues with getting into the system so usability and A11y testing can't be performed.
@stepchud is taking a look and will get back to me when resolved.

Stephen advised to delete browsers cache. Instead, I used an incognito window and that solved the caching issue. Will test that way going forward to avoid the multiple account issues.
@stepchud did suggest holding off on testing this card because there is not way for me to build my list of evaluations until the evaluation builder is ready.

I cannot test this page because the evaluation form builder is not complete. When I click save I get the following error:

@stepchud Testing/review notes:

Evaluation period dates are not displayed
The delete button is implemented, even though it's not needed. Since only one form will be allowed for one challenge/phase and the form will be editable until challenge ends, we agreed that there's no need for delete button.

@r-bartlett-gsa done! ✅

@stepchud Looks great. Thank you!

@r-bartlett-gsa
User Testing Feedback:
AC: If the user does not have any saved evaluations forms, the list is empty

I did not create any forms.  When I went to the Evaluation Forms Page I saw a form:

Confirming that co-managers of a challenge should all see the form that is associated with the challenge?

@mhotch24, this can be a11y tested and responsive tested.

@r-bartlett-gsa I no longer have the ability to check off items. Can you check my permissions please?

In the meantime, I found the following A11y issues:

Screenreader is reading the 'Create New Evaluation Form' button twice in a row. Perhaps because the code should be role="button" rather than nesting a button inside of an .
Table needs a name. 'No Accessibility markup found for this Element'.

@TCKapGrp Your account must be added as a co-manager on a challenge, and that's why you see the form.  I verified that when the user has no forms, an appropriate page is displayed.


chore: add pydantic v2 as dev dependency

Pull Request Checklist

[ ] New code has 100% test coverage
[ ] (If applicable) The prose documentation has been updated to reflect the changes introduced by this PR
[ ] (If applicable) The reference documentation has been updated to reflect the changes introduced by this PR

Description

update the version in pyproject.toml
add pydantic v2 as a dev dependency without which poetry will install pydantic v1 since piccolo has a <2.0 constraint

It looks like this doesn't work because of the other constraints we have.

It looks like this doesn't work because of the other constraints we have.

Hm..maybe pydantic needs to be removed first before installing v1. Let me test the CI with those changes locally.

Maybe this gets rolled into the PDM swap?

@guacs I think we can close this now that we've moved to PDM.  I'm pretty sure it will get installed by default now.  Thoughts?

@guacs I think we can close this now that we've moved to PDM. I'm pretty sure it will get installed by default now. Thoughts?

Agreed.


Validate more asset types

Matching the types supported by the underlying library.

Need to break this issue down into sub-issues (one per file type)


bump tailwindcss to version 0.6.1

Hello and thank you for the awesome template.
This is one of my first pull requests ever.  Please let me know if you need any more information.
Summary:

Updated package.json
reran npx tailwind init
updated README.md to reflect version change

Thanks,
benmp

@allcontributors add @benmp for code


add test ledger workflow

Added workflow to start the ledger and use the ledger browser to check the ledgers state.

We need to decide, which action should trigger this workflow (push / pr /release / manually ...)

fixes #7


Windows 10 Pro Lenovo TrackPoint middle button scrolling not working

Using a Lenovo TrackPoint enable keyboard and wanted to scroll a page in Lagrange v1.1.1, the icon changes into the "scrolling vertically" view, but nothing happens. The viewport does not scroll.

That's interesting. A quick web search suggests that the TrackPoint generates mouse wheel events, but they may not be quite the same as normal wheel events. It's unclear whether they are coming through as usual via SDL.
Are there any system settings for the TrackPoint with regard to scrolling/wheel emulation modes?
Not sure how to go about fixing this... The last time I personally saw a TrackPoint was circa 2004 when I had a ThinkPad as a work laptop. 🙂 I suppose we could try logging all the events received by the app and see what the TrackPoint is sending. Such a logging mode doesn't currently exist, though.

Found a work-around / solution. Disabling the "Thinkpad Preferred Scrolling" makes the TrackPoint scrolling using the middle button work again.
So this issue can be closed as the effort is probably not worth the time.


Include big data experiment in the workflow

This is the draft for an eventual report on the big data set `GrieneisenTSData'. It still needs some text as well as table in the Rmd file and maybe other plots can be added.

I could check once the first full version is ready?


Fix fill_arrayobj

Some fixes:

fill_arrayobj() takes sig, args and also fill_value, this duplication is not needed, since the value is supposed to be in args already.

In the fill_arrayobj(), we check if the fill_value is float by probing args[1]. But when there is no fill_value in args, we are testing on a wrong argument, this happens in the case of dpnp.ones(), dpnp.zeros() etc. which is not correct. But it doesn't produce any bug or incorrect result, since the fill_value is either 0 or 1.

Now the codegen() functions in all intrinsics are cleaner and streamlined.

[x] Have you provided a meaningful PR description?

[x] Have you added a test, reproducer or referred to an issue with a reproducer?

[x] Have you tested your changes locally for CPU and GPU devices?

[x] Have you made sure that new changes do not introduce compiler warnings?

[ ] If this PR is a work in progress, are you filing the PR as a draft?

@chudur-budur The PR is good to go as is. However, we should explore how to use kwargs for intrinsics.


Add hollow-metadataservice-0.1.0@92fcdbb84bc14d7e8a1372e1d93d39451616…

…33ae ⎈
Note: I realize I should start bumping the chart version with my changes. I will do so next time, but would prefer to avoid another PR and review round-trip on the upstream chart this time.

Closing.


Added reference to on behalf of for NodeJS

@maliksahil : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#sign-off


[RN][GHA] Migrate the Prepare Release workflow

Summary:
This change migrates the prepare_release workflow from CCI to GHA
Changelog:
[Internal] - Migrate from CCI to GHA
Test Plan:
Test on GHA

FYI @cipolleschi I am seeing a new failure on main that seems potentially related to this change: E.g. https://github.com/facebook/react-native/actions/runs/9452422456/job/26035889529


Fix contribution page link in README and CONTRIBUTING files

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Closes #3772


Create Xxxp

What is the mistake in this code
Summary of the Pull Request
References and Relevant Issues
Detailed Description of the Pull Request / Additional comments
Validation Steps Performed
PR Checklist

[ ] Closes #xxx
[ ] Tests added/passed
[ ] Documentation updated

If checked, please file a pull request on our docs repo and link it here: #xxx

[ ] Schema updated (if necessary)

why?


Million fails with Remix SPA mode

What version of million are you using?
2.6.4
Are you using an SSR adapter? If so, which one?
None
What package manager are you using?
npm
What operating system are you using?
windows
What browser are you using?
Chrome
Describe the Bug
Setting up Million with a Remix SPA fails. Here is the sample code:
import { unstable_vitePlugin as remix } from "@remix-run/dev";
import { defineConfig } from "vite";
import million from "million/compiler";
import tsconfigPaths from "vite-tsconfig-paths";

export default defineConfig({
  plugins: [
    million.vite({
      auto: true,
    }),
    remix({
      unstable_ssr: false,
    }),
    tsconfigPaths(),
  ],
  server: {
    port: 3000,
  },
});

The error received is:
[vite] Error when evaluating SSR module virtual:remix/server-build: failed to import "/app/routes/_app.sites.$x.$y.$z.$ab.details/route.tsx"

What's the expected result?
It should integrate with Remix SPA same way it does with a Vite+ React SPA and work.
Link to Minimal Reproducible Example
https://remix.run/docs/en/main/future/spa-mode
Participation

[ ] I am willing to submit a pull request for this issue.

You could try setting server: true as well, but, then again a reproduction of the issue would as well be helpful.


[PW-3621] Custom fields for e.g. transaction ID not stored?

Describe the bug
It appears the custom fields like PSP transaction ID are not stored on the order transaction.
Not sure, if that is by design.
Expected behavior
Store custom fields on order transaction.
Also way not store the current PSP reference ID too. why only the original?
Additional context
You execute: $transaction->getOrderTransaction()->setCustomFields($customFields);
However, the Shopware DAL is not an ORM and therefore this does not store the fields at all. You just set it in the order transaction object.
In order to store custom data on the order transaction you need to do something like this:
        $context->scope(
            Context::SYSTEM_SCOPE,
            function (Context $context) use ($transaction, $customFields): void {
                   $this->orderTransactionRepository->update(
            [
                [
                    'id' => $transaction->getId(),
                    // Shopware uses JSON SQL functions to update only provided fields.
                    'customFields' => $cusotmFields,
                ],
            ],
            $context,
        );
            },
        );

Thanks for opening this issue @AndreasA,
I've created an internal ticket to work on it. Your snippet will be useful!
Best,
Ángel

The system scope is most likely necessary as I think you cannot update orders directly otherwise but you can try without it too.


Seg default

When I used LRA to align the HG002 ONT data and ran the following code, I did not get any results. However, other alignment software worked fine.
/home/xxx/00.Software/SVDSS/SVDSS_linux_x86-64 index
/home/xxx/00.Software/SVDSS/SVDSS_linux_x86-64 smooth --reference /home/xxx/01.Reference/Human_GRCh37/GCF_000001405.25_GRCh37.p13_genomic.fa --bam ${sample} --threads 20 > ${path}/${name}.smoothed.bam
    samtools index -@ 20 ${path}/${name}.smoothed.bam
    /home/xxx/00.Software/SVDSS/SVDSS_linux_x86-64 search --index /home/xxx/01.Reference/Human_GRCh37/GCF_000001405.25_GRCh37.p13_genomic.fmd --bam ${path}/${name}.smoothed.bam --threads 20 >${path}/${name}.txt
    /home/xxx/00.Software/SVDSS/SVDSS_linux_x86-64 call --reference

（svdss) [xxx@hpc01 08.lra]$ /home/xxx/00.Software/SVDSS/SVDSS_linux_x86-64 call --reference /home/xxx/01.Reference/Human_GRCh37/GCF_000001405.25_GRCh37.p13_genomic.fa --bam HG002.ONT_lra.smoothed.bam --sfs ./HG002.ONT_lra.txt  --threads 20 --min-cluster-weight 2  --clipped   >calls.vcf
[2024-08-18 17:58:25.669] [stderr] [info] Loading reference genome from /home/jiaoh/01.Reference/Human_GRCh37/GCF_000001405.25_GRCh37.p13_genomic.fa..
[2024-08-18 17:58:47.035] [stderr] [info] Loading SFSs from ./HG002.ONT_lra.txt..
[2024-08-18 17:59:02.036] [stderr] [info] Loaded 15907313 SFSs from 1768750 reads.
[2024-08-18 17:59:02.099] [stderr] [info] Placing SFSs on reference genome
[2024-08-18 18:01:11.676] [stderr] [info] 107/0/0 unplaced SFSs. 0 erroneus SFSs. 3500774 clipped SFSs.
[2024-08-18 18:01:11.676] [stderr] [info] Clustering 1054888 SFSs..
[2024-08-18 18:01:12.731] [stderr] [info] Maximum extended SFS length: 49162bp. Using separation distance: 54078bp.
[2024-08-18 18:01:13.337] [stderr] [info] Extending 314027 clusters..
[2024-08-18 18:03:18.167] [stderr] [info] Filtered 1247 SFSs. Filtered 214848 clusters. Filtered 39 global clusters.
[2024-08-18 18:03:18.167] [stderr] [info] Calling SVs from 314027 clusters..
Segmentation fault (core dumped)

I would greatly appreciate it if you could help me identify the error！！！！

Hi,
I'll try lra on some fastq I have and I'll let you know.
Did you see any warning or error while running the previous steps (i.e., smooth and search)?
Best,

I tried lra but after smoothing, the samtools index failed
[E::bam_read1] CIGAR and query sequence lengths differ for m64062_200604_115437/45811539/ccs
samtools index: failed to create index for "hg007.lra.smoothed.bam"

I checked that read in the lra bam and I found that it's cigar is: 68=1X4=...18=5D5S. I think that having a deletion followed by a soft clip may create issue in the smoothing step. Can you please check if you have a similar error while indexing the smoothed bam file?
If not, can you please share where I can get the fastq you are using? I think I can replicate the error that way.
Best,

Hi,
the issue is that in some regions (like the one in the screenshot) potential SV are very long.
When SVDSS try to create the consensus (via abpoa), it fails due to not enough memory (or something like that). These cases are not expected with HiFi reads that are way shorter than ONT. The only patch I can think of for now is to filter out those regions. I prepared this patched binary. I tried it on the lra bam and it should work - please try and let me know.
However, I'd like to point out that we developed SVDSS to work with HiFi data and not ONT - although it could work, results may not be the best. We are currently working on extending the approach to ONT, but it's not ready yet.


Rewrite for gitlab CI/CD migration

simplifies the test and build system by moving items that fit the Makefile model out of scripts
makes local testing and verification of changes easier by sharing the commands used by CI
follows the k8s model of putting CI scripts in hack
includes an incidental change to data to simplify golint command in make vet.

BTW: There is a tool gaining traction in the Golang community that drives build, dist creation and the GitHub release into a YAML driven declarative configuration file.
https://goreleaser.com
by adding a go get to make setup and a YAML file we gain a simpler Makefile/build script system.
I haven't used the tool yet.

If I'm reading this right, go build for different os/arch is being done in serial rather than parallel under the change. How does this affect the amount of time a CI run takes?

nice overall!  looking forward to seeing the rest of this.

inre @mattfarina
It is the longest part of the run right now. It will not be when the migration is complete. The build/release will come at the end of the run after all significant testing is over so we may not notice or care.
That said. Were we to use the goreleaser I mentioned someplace in this conversation we would get parallel builds as we had with gox.

Here is an example of a prominent Golang project using goreleaser (although there use is more sophisticated than we are likely to need in the near term).
https://github.com/gohugoio/hugo/releases

Ready for prime time: @samsung-cnct/commontools

This is ready. I’ll have a new pull request that does a snapshot build using gorealaser and a make task for humans doing a release.

Switch master branch protection to gitlab.

strange how the in container tests at gitlab fail while all local tests pass. first vet fails and unit test passes, then after a change to vet, unit test begins failing.

Found it in shared setup (that wasn't really shared(.

:rofl:


Extend Get-Command to support a -Resolve switch for resolving to an alias' underlying command

It would be convenient  to allow passing -Resolve  to Get-Command so as to immediately display information about the command underlying an alias, if applicable.
In other words: given an alias name, -Resolve would act like (Get-Command someAlias).ResolvedCommand.
This is more useful than the default behavior, where the output for an alias name only indicates the name of the underlying command.
For non-alias command names, the switch should have no effect.
Desired behavior
PS> Get-Command gci -Resolve  # wishful thinking; equivalent of (Get-Command gci).ResolvedCommand

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Cmdlet          Get-ChildItem                                      <IP_ADDRESS>    Microsoft.PowerShell.Management

Environment data
Written as of:
PowerShell Core 6.1.0-preview.4

Great idea !!!

I think it could be -ResolveAlias.

@iSazonov:
I was thinking -Resolve for consistency with other cmdlets (currently only Join-Path and Split-Path), but I guess we can do both - make one an alias of the other.
(Even though dynamic syntax would allow using -Resolve in lieu of -ResolveAlias anyway, it's better to "make it official".)

I suggest -ResovleAlias because it is self-documented and it is not clear that -Resolve should resolve - full module name, full source path and so on.

@iSazonov:
Maybe we can turn that into a virtue, given that there's at least one other way to resolve a command name:
For external programs that happen to be symlinks, make -Resolve show the ultimate target file path (at least currently, such a feature is of much more interest on Unix-like platforms); e.g.:
# Currently, you get just the symlink's path
PS> Get-Command pwsh

CommandType     Name       Version    Source
-----------     ----       -------    ------
Application     pwsh       <IP_ADDRESS>    /usr/local/bin/pwsh

# Wishful thinking: -Resolve shows the symlink's (ultimate) *target* path
PS> Get-Command pwsh -Resolve

CommandType     Name       Version    Source
-----------     ----       -------    ------
Application     pwsh       <IP_ADDRESS>    /usr/local/microsoft/PowerShell/6.1.0/pwsh

On a side note, complementing -Resolve with something like -Follow or -ResolveChain  may make sense, which could emit both the immediate target and the (chain of) ultimate target(s), as a sequence of objects.
For instance, with gci you'd get both the alias info and the underlying cmdlet info, and with pwsh you'd get the original symlink info as well as info for their target(s).

@iSazonov : Too late - I've fallen in love with the idea 😜.
I can tell you that inspecting the true filesystem locations of symlinked executables is something I deal with frequently (on Unix-like platforms).
Format-List would only help if you crammed more information into the one and only output object currently emitted, which strikes me as a bad idea.
This issue is about resolving to the command that is effectively used / to see the chain of commands (symlinks) that lead to it.
I've updated the original post with my suggestions; let's see what others think.


File router cannot handle view that have same names

Describe the bug
I created two views that have same names but different paths. As you can see below, they are listing in available routes but one of them returns 404

Both have exported syntax like this(not sure if it is related though):
const Overlay = () => {
...
}
export default Overlay;

Expected-behavior
Should handle both views in different paths. /overlay and /overlay/overlay should work.
Reproduction
Create the same structure in an example project and you would get the same result
System Info
Flow version: 24.4.0.alpha31
Platform version: 24.4.0.alpha24

Does this work if you move overlay.tsx to be<EMAIL_ADDRESS>

Does this work if you move overlay.tsx to be<EMAIL_ADDRESS>
Then localhost:8080/overlay works but localhost:8080/overlay/overlay does not. It should work because I have two separate files for both routes

https://github.com/vaadin/hilla/assets/100126447/9d62ddee-ea31-4dbc-bb8e-bc94d576a417

Also, the menu has 2 overlay links

What could cause this is that we match the /overlay directory route, and then the Flow fallback index in that directory route also matches.
If we give less priority to directories over files when sorting FS route entries, it would solve this issue.


Fix command-line examples

Description of changes:
Some of the command-line examples to upload an artifact to a maven type repository were incorrect. Also when publishing the package it is required to tell AWS CodeArtifact in which namespace (ie group in maven parlance) it is, or AWS CodeArtifact won't find it.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Thanks @masterzen ! We always appreciate contributions to the docs, getting this update in.


Add no-std Support for Plonky2 and Starky

This PR adds makes the plonky2, starky, and associated gadget libraries no_std compatible.
Structure of Changes
For the most part, the changes involve only adding no_std-compatible versions of the existing data structures (and importing from alloc) and restructuring the feature-selection settings for each crate. However, in some places there are some more specific changes, mostly in the following two areas:

Serialization in plonky2
Inline Assembly Support in plonky2_util

Serialization
For serialization, we noticed that Cursor<Vec<u8>> is used as the canonical reader/writer for plonky2 objects. However, there is no no_std equivalent for Cursor without possibly importing a crate. Instead, the serialization methods on the Buffer type were generalized to a Read and Write trait. For writing, a Vec<u8> is enough, and we get the added benefit that it is a completely infallible operation so we can remove the error handling in that case. For reading, we still need a Cursor<Vec<u8>> so the Buffer object still implements Read and Write. In this case, Buffer was delegated to the std-only feature set. In future work we may consider finding a canonical writer that is no-std compatible.
Inline Assembly
In plonky2_util inline assembly is used to provide a custom compiler hint:
https://github.com/mir-protocol/plonky2/blob/ea7fbed33a8d132ecc4dc92e9ce49639fdfe3753/util/src/lib.rs#L267-L273
To make this no-std compatible we add a target selection at compile-time that only selects the targets that support inline assembly as of the authoring of this PR.

Thank you for the contribution!
Could you explain the motivation a bit? I think it would be interesting to have plonky2 running on targets like wasm, but my understanding is that no_std is not required there, just certain std methods are unimplemented. Similarly for Solana, my understanding is that no_std is not required, although certain things like std::net are missing.
Is there a use case that requires this, or do you see it as more of a best practice, to enforce certain compatibility guarantees?

It's a bit of both.
Building crates as no-std first and then extending by std as an optional feature is the best practice (unless the purpose of the crate is to do IO, networking, ... etc.). Even though WASM may work with std in some cases, unless you know exactly which APIs are unsupported you will get spurious runtime errors (instead of compile-time ones). In resource-constrained environments, sandboxed environments, or environments that use a subset of WASM like in substrate-based blockchains, very strict adherence to no-std is actually required. Also, moving to a no-std compatible build makes the plonky2 crate much friendlier to the rest of the Rust ecosystem by allowing other no-std crates to safely import code independent of whatever reasons they may have for being no-std.
One other benefit of moving to a minimal dependency implementation (I'm including the standard library as a dependency here) is that we can see exactly where we introduce a dependency and its usage scope.

Thanks for the additional context; seems reasonable to me.
For future changes I'd prefer if you could find a way to split things up, ideally with uninteresting changes (like the alloc imports) grouped together for ease of reviewing. I've been reading through this and it looks good so far, but still a ways to go. I'll try to finish fairly soon so we can get this in before a lot of conflicts come up.

Looks good otherwise!

Thanks for the PR comments!

For future large changes I'd prefer if you could find a way to split things up, ideally with uninteresting changes (like the alloc imports) grouped together for ease of reviewing.

Do you mean in another PR or in my commit structure? Wasn't sure what the policy was for this repo so I just did it all together. Let me know and I'll use whatever standard you want to use for future PRs, happy to do that.

And we usually avoid rearranging imports, since there isn't really a standard layout (VSCode and IntelliJ defaults differ), although maybe we could enable some rustfmt options to address that.

I had noticed that most imports were using the same declaration patterns but there were some minor anomalies so I just corrected those. And also I found that imports_granularity = "Module" matched the majority convention that seemed to be used in the code, so I added that to rustfmt.

Thanks, I think this is good to merge, looks like there's just a few conflicts if you wouldn't mind resolving those?
Re PR flow, ideally we like separate PRs for repetitive diffs, like rustfmt changes or things that touch a lot of imports. Just like to get those merged quickly with minimal conflicts, while taking our time to review and discuss the more interesting changes. I understand it's not always easy to break things up though.


https://codesandbox.io/s/6j1760jkjk example is not going down in the counter when removes element

https://codesandbox.io/s/6j1760jkjk
Solution:
Change removeFriend method for the next one:
const removeFriend = index => () => {
setIndexes(prevIndexes => [...prevIndexes.filter(item => item !== index)]);
setCounter(prevCounter => prevCounter - 1);
};

good point! thanks fixed.


Try using XStream as classloader state cloner for the PowerMockRule

From<EMAIL_ADDRESS>on October 17, 2010 22:03:09
There are several problems with the current PowerMockRule which seems hard to fix (the JVM crash for example). An alternative approach would be to try out XStream to do the serialization and deserialization of Object between classloaders.
Original issue: http://code.google.com/p/powermock/issues/detail?id=286

From<EMAIL_ADDRESS>on April 10, 2011 05:53:01
Status: Fixed


chore: add more tests to ListObjects

Given a ListObjects query for user u and relation r and type o and response with objects o1 and o2: Check(u, r, 01) and Check(u, r, o2) should return true

Codecov Report
Patch coverage: 100.00% and project coverage change: +0.03 :tada:

Comparison is base (5472a1d) 84.43% compared to head (5e7be0e) 84.46%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #573      +/-   ##
==========================================
+ Coverage   84.43%   84.46%   +0.03%
==========================================
  Files          87       87
  Lines       13607    13647      +40
==========================================
+ Hits        11489    11527      +38
- Misses       1740     1742       +2
  Partials      378      378

Impacted Files
Coverage Δ

pkg/server/test/list_objects.go
77.70% <100.00%> (+1.67%)
:arrow_up:

cmd/run/run.go
61.22% <0.00%> (-0.41%)
:arrow_down:

internal/graph/check.go
90.18% <0.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Made as part of https://github.com/openfga/openfga/pull/599


Make menuItem accessible for keyboard inputs

Thanks for the library, we are enjoying it!
Because you use an MUI Menu with MenuItem, I would have expected the following behavior.
Pressing the letter "B" on my keyboard while the dropdown is open should go to the first country that starts with the letter B.
This is the default behavior of MUI and I'm not exactly sure why it isn't working here. https://mui.com/material-ui/react-menu/#main-content
Steps to reproduce
Open the flagsMenu, press a letter of the country that you are looking for, and nothing happens.
I tried debugging it myself, but couldn't get it to work fully...
When placing a dummy <MenuItem /> right above the <FlagsList .../> (so it's the first element) then it did work. 🙀

FYI: I have opened an issue at MUI: https://github.com/mui/material-ui/issues/35255
The problem is fixed when using  directly instead of deeply nesting your components.
Simplified example that does work:
 <Menu
      anchorEl={anchorEl}
      open={Boolean(anchorEl)}
      {...rest}
    >
      {countriesFiltered.map((isoCodeItem) => {
        return (
          <MenuItem onClick={handleClick} key={isoCodeItem}>
            {isoCodeItem}
          </MenuItem>
        );
      })}
    </Menu>


vscode extension: formatter adds a trailing new line on every invocation

I have VSCode set up to format a file upon save, so this means that every time I save an edited *.spvasm assembly file a new line is added to the end of the file, even if there are several there already. This is on Windows, so I'm using \r\n line endings, in case that's relevant.

I believe this was fixed with https://github.com/KhronosGroup/SPIRV-Tools/pull/4246


Foundry v11 Compatibility

Can we expect an update soon? I'm currently running a game in Foundry with a few others in different systems, and I'd like to install the updates for those, as well as any modules that have been updated.

Are you aware of any specific compatibility issues, or just asking when the system will be flagged as v11 compatible?
Not sure exactly when I'll put out a new package, but in limited testing I've not seen any issues with the system in v11.

I've been holding out on upgrading for fear that my campaign may break. I was of the understanding (erroneously, I guess) that the DB change in the new version would cause more problems than it seems to be. I'll keep an eye out here and see if anyone brings up any issues, and upgrade eventually if nothing pops up.
Thanks!

One thing I noticed is that the "Apply Skill Improvement" button seems to only work for one skill in v11. It rolls for all of them, but only one gets unchecked and increased. Clicking it n times does the trick though!

@tompudding Looking at it, kind of surprised the behavior on this would be any different in v10 than v11. But I think I know how to resolve it.
@jalensailin I will make a change to _applySkillImprovements() so that all the skill updates to the actor happens in one call synchronously (as opposed to doing each skill individually in the loop). In my v11 test environment, it seems to work.
I've seen issues like this before where because multiple updates are all happening async, only the last update goes through and wipes all the changes before it. If you think there is a more elegant way to do it though, feel free to modify it.

@TheLastScrub
Feel free, thanks for the update and apologies for introducing an error!

@heliobolseiro it is currently not possible. can you open up an actual issue for this, rather than just a comment on an unrelated issue? Currently it isn't possible to translate fully. we would need to make it so that all of the strings are translatable. this would require some work on our end. can try to let you know when it's ready


[TF-256] fix deserialization errors when differentiating wrt subset of params

Adds parameter differentiability to SILFunctionType serialization.
Clears parameter differentiability when constructing the original SILFunctionType from a differentiable SILFunctionType. (Also adds an assertion to catch it earlier if we forget in the future).
Adds a test.

This should fix https://bugs.swift.org/browse/TF-256.

@swift-ci please test tensorflow


MINOR: adding schema registry configs

Added configs for Schema Registry in an effort to see if this helps with examples tests flakiness since most of the failure involve Schema Registry

Increased store init timeout to 90 seconds
Increased store operation timeout from 500ms to 10s
Set debug flag to true

merged with pint


fix-deprecated-currentContext

application.android.currentContext has been removed in NativeScript 6
So let's remove usage here as well.
Use:
(application.android.foregroundActivity || application.android.startActivity)
or
application.android.context
instead.

Thank you!

Not Autherized how to solve it?

The reason I didn't publish this before is because during testing I ran into runtime permission issues which I wanted to tackle, but since folks are now asking me why it's taking so long I decided to publish it as is. Thanks for the PR!


Can I detect the language when I change the pathname (detect via path)?

🚀 Feature Proposal
I've detected the language by the path order. But when I changed the path, the language does not change. I don't want to use the changeLanguage API. I want to sync the language with the browser when I change the URL.
This is my config
 detection: {
      order: ['path'],
      lookupFromPathIndex: 0,
      checkWhitelist: true,
 }

Example
Step by step

Start with http://domain_name/url (this is default lng)
Change the URL to another URL (http://domain_name/en/another-url)

Works already...if working correctly at all -> if i18next.init runs it sets the language based on detection - if there is no other detection beside path and you do not set cache to eg. cookie or localStorage it will pick up language from path as usual...
you might have to set: caches: [], in your detection options to not cache the first detection

@jamuhl It still not working. It just detects in the first-time when setup initial i18next.
But, I think you've confused. I mean, change the URL inside App (Using react-router-dom to redirect to another URL. Not reload).

@nguyenvantruong17 I think you're confused 😉...the detector does not know anything about react-router...and runs only on init...so you will need to call changeLanguage beside navigating to new language on clientside

@jamuhl Thanks for your help. I'll close this issue.


ZBL empirical correction is broken

When a ZBL empirical correction is included, model initialization crashes with the following error
TypeError: Gradient only defined for scalar-output functions. Output had shape: (1,).

Example model configuration:
model:
  empirical_corrections:
  - name: zbl
    r_max: 1.5
  # ...

The exponential empirical correction works fine.

thanks, I'll look into it

fixed by #362

should be fixed now on main


Integrating into Kubescape network Scanner using the Package structure

Fixes #2
( This is a WIP PR )

Defined project layout

├── LICENSE
├── README.md
├── cmd
│   └── main.go
└── internal
    └── pkg
        └── network-scanner
            ├── core
            │   ├── interface.go
            │   └── types.go
            └── modules
                ├── portdiscovery
                │   └── portdiscovery.go
                └── servicediscovery
					└── al_mysql.go
					└── so on.........

Integrated portdiscovery in network Scanner
Integrate servicediscovery in network scanner ( #TODO )

@slashben Could you check if this layout is appropriate and aligns with kubescape?

@Bezbran can this project layout work?

Hi @0xquark, can you please ellaborate about the internal/pkg/network-scanner path? Why not put the content of this path on the root of the repository?
You can take a look on kubescape repo, it seems well organized.

Hi @0xquark, can you please ellaborate about the internal/pkg/network-scanner path? Why not put the content of this path on the root of the repository? You can take a look on kubescape repo, it seems well organized.

Hi @Bezbran ! The internal/pkg/network-scanner was chosen by ben ( CC : @slashben  ) for which i believe the right reason was to having a separate internal/pkg directory is to separate internal packages that are not meant to be exposed to users or other packages outside of the project.

Hey @slashben ! Can you review the changes in an hour? I am making some final fixes then this PR would be ready to review

@0xquark working on it!

@slashben For CLI : https://github.com/spf13/cobra
This might be helpful....I've seen it used in many big projects

@0xquark sure, my code is just an example!
The point is that I try to sort out the directory structure, give me 30m

Looks ok for me!

This PR is ready to be Merged 🚀


IREE_SIZE_OPTIMIZED defines NDEBUG on MSVC, and not on other toolchains

The discrepancy is the bug. IREE_SIZE_OPTIMIZED should consistently either imply NDEBUG or not.
There is the possibility that some header libraries (maybe even the C++ standard library in some implementation) will have symbols whose definition differs depending on NDEBUG; in that respect, to be safe, it would be best to resolve this by never manually defining NDEBUG, letting the CMAKE_BUILD_TYPE being the only global control enabling/disabling this. Though, we probably also have that problem with IREE_ENABLE_ASSERTIONS... I may just have stepped into a bigger trouble than I was bargaining for here.
The immediate motivation was to have something to link to from a review comment on #14216...

Duplicate of https://github.com/openxla/iree/issues/9686 ?

The spirit of the issue I linked is: IREE_SIZE_OPTIMIZED was only implemented on MSVC. The other platforms have placeholder behavior and need someone who does enough development on those platforms to determine what the option should do.

+1 - messing with NDEBUG outside of CMAKE_BUILD_TYPE is a recipe for trouble.
I'd rather that if using this flag, we issue a warning if assertions are in fact enabled.


[build-script] Add an incremental linux asan preset called buildbot_incremental_linux,asan.

[build-script] Add an incremental linux asan preset called buildbot_incremental_linux,asan.
Want to use it to look at a bot failure in a normative way. We should really get this on one of the bots at some point.

Just tried it on a side machine I have. Seems to work! It is firing on some leaks in Generic Signature Builder... ; ).
@swift-ci Please smoke test and merge


[Feature Request] @inboxkitten.com emails marked as malicious for major social media sites

So I've been testing with fake-mails on Discord, and when I tried this, it gave me 2FA lockout. Now, unfortunately, I was testing on my main alt so now I'm locked out with that account. Is it possible if we could change the domain name to a few set choices?
Example:
@inboxkitten.com
@kittensinbox.net
@theinboxofkittens.org
@kittensrus.com

Or pony up forty-four cents for your own domain?

wait what? Oh... I didn't even know those existed

If you purchase any domain - and point it to inboxkitten - it will probably work.
Alternatively, purchase a domain and deploy a copy of this server for use =)
Will gladly accept any sponsored domains to point to this.


bot4s.telegram 4.0.0-RC1 is out!

The project went through a major refactor, some changes/additions :

Repo moved to bot4s organization
Got a community-friendly domain: bot4s.com
Scala.js support via sttp, run your bots on the browser!
Library comes in two artifacts: core + akka
Several backends: akka-http, scalaj-http and sttp (incomplete implementations for hammock and RosHttp due to limitations)
Ported all JSON serialization from json4s to circe (transparent for users)
Support for API v4.0
Use mill instead of SBT
Support for proxies (using ScalajHttpClient).

To try the latest version add to your build.sbt:
resolvers += Resolver.sonatypeRepo("staging")

libraryDependencies ++= Seq(
  "com.bot4s" %% "telegram-core" % "4.0.0-RC1",
  "com.bot4s" %% "telegram-akka" % "4.0.0-RC1"
)

Hey. The main readme in repository does not mention that you need staging repository, leading to confusion :) That should be fixed.


feat/Use local model for hi_res partition

Hello,
Maybe this feature already exist but I didn't manage to implement it.
I work on a network that blocks huggingface and I would like to run:
elements = partition_pdf(filename=PDF_PATH, strategy='hi_res', infer_table_structure=True)
But the function cannot run because it's trying to access the yolox model on the hub:
SSLError: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /unstructuredio/yolo_x_layout/resolve/main/yolox_l0.05.onnx (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))"), '(Request ID: 757ef56e-88d9-4a7a-88ef-ff3fade2139c)')
My question is: If I manage to download the model on my machine somehow, how can use it with the ustructured library without having to call the https request ?
I hope my explainations are somehow ok.
Thanks in advance.

Sorry I don't have an answer but I'd like to ask, is it trying to download the model files or do inference elsewhere? Also saw a page from unstructured io docs that might help https://unstructured-io.github.io/unstructured/installation/full_installation.html#setting=up-unstructured-for-local-inference

It's trying to download the model from hf yes it's the default behaviour or the unstructured library.

I have encountered the same issue.
I explored the source code a little bit and found these lines of  code in site-packages\unstructured_inference\models\yolox.py (line 34 -59):
MODEL_TYPES = {
    "yolox": LazyDict(
        model_path=LazyEvaluateInfo(
            hf_hub_download,
            "unstructuredio/yolo_x_layout",
            "yolox_l0.05.onnx",
        ),
        label_map=YOLOX_LABEL_MAP,
    ),
    "yolox_tiny": LazyDict(
        model_path=LazyEvaluateInfo(
            hf_hub_download,
            "unstructuredio/yolo_x_layout",
            "yolox_tiny.onnx",
        ),
        label_map=YOLOX_LABEL_MAP,
    ),
    "yolox_quantized": LazyDict(
        model_path=LazyEvaluateInfo(
            hf_hub_download,
            "unstructuredio/yolo_x_layout",
            "yolox_l0.05_quantized.onnx",
        ),
        label_map=YOLOX_LABEL_MAP,
    ),
}

So I guess the models from huggingface are hard-coded to download from the url at the moment.
Could we have the feature to config the models to some pre-downloaded path in the later version?
Thanks!

Not sure if it will help, but have you tried specifying the custom model (undocumented): https://github.com/Unstructured-IO/unstructured/pull/2462

Hey I haven't made any progress on this unfortunatelly I'm lost in the module code...
There's probably a modification to do in the get_model() function on the unstructured-inference dependency.
The best implementation to me would be to simply pass the a new argument: "hi_res_model_path" to the partition_pdf function:
elements = partition_pdf(filename=PDF_PATH, strategy='hi_res', hi_res_model_path='/path/to/model', infer_table_structure=True)
Anyone able to evaluate to work needed to develop this ?

I encountered the same problem too

I finally made it work locally with these changes.
Depending on the setting you are using you need give local path as one of the input . You can download the models for once, and then save them, or just download them directly and give their path to it.
1)env/lib/python3.9/site-packages/unstructured_inference/models/tables.py
logger.info("Loading the table structure model ...")
model_path = 'path to table_transformer_recognition'
self.model = TableTransformerForObjectDetection.from_pretrained(model_path, use_pretrained_backbone=False)
self.model.eval()
#self.model.save_pretrained("path to save model")#this one can save the model the first time by getting connected to hub.( if you don't want to save the model directly you can use it once to download the model and then comment it.

In case you are using yolox in the setting, then you need to change this one
2) env/lib/python3.9/site-packages/unstructured_inference/models/yolox.py
MODEL_TYPES = {
    "yolox": LazyDict(
        model_path='path to yolox_l0.05.onnx folder downloaded from hugging face',
        label_map=YOLOX_LABEL_MAP,
    ),

Hope it can solve your issue

Thanks all, we'll plan to add better support for this.

Same problem. Any progress there?

I finally made it work locally with these changes. Depending on the setting you are using you need give local path as one of the input . You can download the models for once, and then save them, or just download them directly and give their path to it.
1)env/lib/python3.9/site-packages/unstructured_inference/models/tables.py
logger.info("Loading the table structure model ...")
model_path = 'path to table_transformer_recognition'
self.model = TableTransformerForObjectDetection.from_pretrained(model_path, use_pretrained_backbone=False)
self.model.eval()
#self.model.save_pretrained("path to save model")#this one can save the model the first time by getting connected to hub.( if you don't want to save the model directly you can use it once to download the model and then comment it.

In case you are using yolox in the setting, then you need to change this one 2) env/lib/python3.9/site-packages/unstructured_inference/models/yolox.py
MODEL_TYPES = {
    "yolox": LazyDict(
        model_path='path to yolox_l0.05.onnx folder downloaded from hugging face',
        label_map=YOLOX_LABEL_MAP,
    ),

Hope it can solve your issue

Thank you for posting, this worked for me as well!

Hii all !
I am facing the same error due to the corporate proxy. I need to extract table and images out of pdf files but since unstructured is trying to call the hugging face it fails to do so. I have downloaded yolo_x_layout/yolox_l0.05.onnx model from the hugging face and I tried to change the following.
C:\Users\Username\AppData\Local\Programs\Python\Python312\Lib\site-packages\unstructured_inference\models\yolox.py:
 "yolox": LazyDict(
        model_path=r"C:\Users\Username\Desktop\Models\yolo_x_layout\yolox_l0.05.onnx",
        label_map=YOLOX_LABEL_MAP,
    )

I am facing the following error:
Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from C:\Users\k.divyanshu\Desktop\Assignment_3\Models\yolo_x_layout
failed:system error number 13

Can someone let me how to pass the downloaded model correctly in order to correctly extract the table and images from a given PDF File.

@kd10041 If you encounter the same issue, you can use the following workaround. Execute these steps in the directory where you run the script:
mkdir -p unstructuredio/yolo_x_layout/
wget --no-check-certificate https://huggingface.co/unstructuredio/yolo_x_layout/resolve/main/yolox_l0.05.onnx -O unstructuredio/yolo_x_layout/yolox_l0.05.onnx

Alternatively, you can download the file manually from the following link:
Download yolox_l0.05.onnx
After downloading, copy the yolox_l0.05.onnx file to the unstructuredio/yolo_x_layout/ directory:
cp path/to/downloaded/yolox_l0.05.onnx unstructuredio/yolo_x_layout/


vt_argopt: when using allocator and passing in a char*=NULL it avoids allocator management due to using strdup()

line 206 and onward: case VT_TYPE_CSTR


Consolidate graphValidationState and validationState in submission and metadata entities

User Story
As a wrangler/contributor, I want to easily see if an entity is invalid because of graph validation or metadata validation.
At the moment, the state of an entity (biomaterial/process/file/protocol) will only show "Valid"/"Invalid"/"Draft" etc when the metadata is of that state. It does not consider graph validation. It should show either "Graph Invalid" or "Invalid" when graph is invalid too.
Also at the moment, the submission should show "Invalid" or "Graph Invalid" when graph validation has failed.

Can also improve error reporting when this is done to show graph validation success messages alongside the valid entitites. Something like "This entity is not causing an invalid graph".

Hi @jacobwindsor , I'm just wondering if this task is based on the graph validator as it is on EC2, or whether it has been or is in the process of being integrated into ingest?

@ami-day no, this has nothing to do with the graph validator on EC2. Besides, I don't think you need to use that anymore

@jacobwindsor to work on this

Assigning @aaclan-ebi and @amnonkhen as primary code reviewers
https://github.com/ebi-ait/ingest-state-tracking/pull/9
https://github.com/ebi-ait/ingest-core/pull/102
https://github.com/ebi-ait/ingest-ui/pull/143
https://github.com/ebi-ait/ingest-graph-validator/pull/60
https://github.com/ebi-ait/ingest-integration-tests/pull/20

Testing this will involve the following:
For a metadata valid & graph valid submission

Upload the submission
Ensure the state shows "Metadata Valid"
Ensure you can navigate to the "Validate" tab
Trigger validation and watch the transition go between "Graph Validation Requested", "Graph Validating", "Graph Valid"
Ensure the validationState of the project is "Valid" (see in API)
Ensure you can submit

Metadata invalid

Upload the submission
Wait for validation and see "Metadata Invalid"
Ensure you cannot navigate to the "Validate" tab

Metadata valid & graph invalid

Upload the submission
Wait for validation and "Metadata valid" state
Navigate to "Validate" tab and trigger validation
Ensure you see transitions between "Graph Validation Requested", "Graph Validating", "Graph Invalid"
Ensure you can navigate to the error causing entities and see the detailed messages

Other checks

All submissions page works as it always has
My projects page works as it always has
You can filter by validation states (e.g. "Invalid") in the submission page

Deployment order

ingest-graph-validator

[ ] dev
[ ] staging
[ ] prod

ingest-core

[ ] dev
[ ] staging
[ ] prod

ingest-state-tracking (check logs for failures after deploying)

[ ] dev
[ ] staging
[ ] prod

ingest-ui

[ ] dev
[ ] staging
[ ] prod

Waiting for review.

Waiting for review. @MightyAx will take a look today.

Done with review.
@jacobwindsor merge it!

@jacobwindsor to test after stand-up and fix today. @ESapenaVentura to test afterwards

There is a PR for the UI, core, state tracker.
There is a related ticket has been created (#578).

@ESapenaVentura to review alongside #578 and #517

Still testing this

Going with the tests (The submission was generated with the data here https://github.com/HumanCellAtlas/schema-test-data, submission here https://staging.contribute.data.humancellatlas.org/submissions/detail?uuid=558f1636-0ae9-4747-9238-6276f14aa563&project=218a4231-28a0-46f4-b174-8b9901986e98):
Metadata valid, graph invalid

Works as intended!! States transitioned from valid metadata, to graph validation requested, to a couple of other steps until "invalid graph"
Clicked on the hyperlink and showed me only entities that cause invalid graphs

Metadata valid, graph valid

After the last step, I corrected the issue (protocols were not connected to anything)
I found some issues (reported below) regarding correcting, but once corrected I hit validate graph and worked fine!
Tried to cause graph changes (valid) by deleting 2 files to test #517

Submission went back to draft, so #517 is working as intended!

Metadata invalid, graph valid

Changed a couple of fields on donor AP1, saved and ingest re-validated without any issue, pointing out to the proper biomaterial

Metadata invalid, graph invalid

Deleted a link to a file to generate an orphaned entity while metadata was still valid and requested graph validation
Modified AP1 donor to have invalid metadata
Metadata status overrides graph validation status (expected) and the errors are shown for both (expected), so great!!

Discovered bugs/weird things noticed (may not even related to the changes)

After clicking on the hyperlink to show the graph errors, changing the filter to blank takes a very long time to be effective (I was actually writing this down as a bug until I noticed it fixed)
Service unavailable error when trying to search for the protocols to link them back, wasn't working and threw me out of the page (but I was still logged in)
Protocols page takes ages to load
When fixing the linkings, suddenly the submission went back to a state of "metadata valid" before hitting the validate graph again. I suspect this has to do with adding entities to the process from other submisssions (I added protocols that were named the same)
Throws "service unavailable" error intermittently, I checked and I wasn't logged out so I don't know why this is happening

Other comments

I like the greyed out button, but maybe it could change the text? something like "Validation requested" or something like that?


Missing SEQRES Information in seq_db

I used the following code to get the sequence information from seq_db.
from pinder.core.index.utils import get_sequence_database
from pinder.core import get_index
import pandas as pd

seq_db = get_sequence_database()
seq_index = get_index()

However, I noticed that seq_db seems to provide sequences only for structured residues (i.e., residues with 3D coordinates). Residues listed in the SEQRES section of the PDB, which represent the full chain sequence(and also include the missing residues in the structure), appear to be excluded.
Q1. Does seq_db only include structured residues?
Is there a way to retrieve the full SEQRES sequence using the pinder library?
Q2. How are residues numbered in Pinder PDB files?
Are they aligned with the SEQRES sequence (including missing residues)?
For example, can I assume that residue number 4 in a Pinder PDB corresponds to the fourth residue in the SEQRES sequence?

Hi @tpdmskim ,

Q1. Does seq_db only include structured residues?
get_sequence_database() does only included structurally resolved residues.

Is there a way to retrieve the full SEQRES sequence using the pinder library?
Yes! There are two potentially useful supplementary metadata for your use-case:

entity_metadata
chain_metadata

entity_metadata
entity_metadata contains next-gen mmcif extracted entity metadata, which should be equivalent to the SEQRES records you are referring to:
from pinder.core import get_index, get_supplementary_data
entity_meta = get_supplementary_data("entity_metadata")
print(entity_meta.head(1).T.to_markdown())

0

entry_id
7rw0

entity_id
1

sequence
SGFRKMAFPSGKVEGCMVQVTCGTTTLNGLWLDDVVYCPRHVICTSEDMLNPNYEDLLIRKSNHNFLVQAGNVQLRVIGHSMQNCVLKLKVDTANPKTPKYKFVRIQPGQTFSVLACYNGSPSGVYQCAMRPNFTIKGSFLNGSCGSVGFNIDYDCVSFCYMHHMELPTGVHAGTDLAGNFYGPFVDRQTAQAAGTDTTITVNVLAWLYAAVINGDRWFLNRFTTTLNDFNLVAMKYNYEPLTQDHVDILGPLSAQTGIAVLDMCASLKELLQNGMNGRTILGSALLEDEFTPFDVVRQCSGVTFQ

length
306

organism
Severe acute respiratory syndrome coronavirus 2

tax_id
2697049

type
polypeptide(L)

asym_id
A

pdb_strand_id
A

chain
A1

uniprot
P0DTD1

length_resolved
306

This was derived via this method in ingest:
https://github.com/pinder-org/pinder/blob/20487ac403516f453d8be1b0bdc07a9f7a77d9df/src/pinder-data/pinder/data/get_data.py#L325-L414
chain_metadata
This contains pinder-extracted, residue-level information for different residue numberings and is condensed into comma-separated values per-chain. Gaps are mapped to empty strings. This was used during other metadata/annotation stages, where e.g., the ECOD domain was available in author-numbering or author chain naming, but the pinder structures are numbered sequentially from 1. See here for an example of how this info was consumed to extract/calculate portions of the dimer chain:
https://github.com/pinder-org/pinder/blob/20487ac403516f453d8be1b0bdc07a9f7a77d9df/src/pinder-data/pinder/data/get_dimers.py#L1214-L1257
this is what it looks like:
from pinder.core import get_index, get_supplementary_data
chain_meta = get_supplementary_data("chain_metadata")
print(chain_meta.head(1).T.to_markdown())

0

id
6rwh__A1_P31947--6rwh__A2_P31947

asym_id_R
A

asym_id_L
A

entity_id_R
1

entity_id_L
1

pdb_strand_id_R
A

pdb_strand_id_L
A

chain_R
A1

chain_L
A2

resi_R
1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253

resi_L
1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253

resi_pdb_R
-4,-3,-2,-1,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248

resi_pdb_L
-4,-3,-2,-1,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248

resi_auth_R
-4,-3,-2,-1,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,,,,,,,,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,,,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,,,,,,,,,,,,,,,,,

resi_auth_L
-4,-3,-2,-1,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,,,,,,,,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,,,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,,,,,,,,,,,,,,,,,

resi_uniprot_R
,,,,,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,,,,,,,,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,,,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,,,,,,,,,,,,,,,,,

resi_uniprot_L
,,,,,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,,,,,,,,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,,,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,,,,,,,,,,,,,,,,,

Here, the columns mean:

resi_L/R -> residue numbering in pinder PDB files
resi_pdb_L/R -> residue numbering in RCSB/PDB numbering
resi_auth_L/R -> residue numbering defined by author
resi_uniprot_L/R -> residue numbering in uniprot sequence, when uniprot is defined.

In addition to the above, we make un-compressed per-chain residue mapping information accessible (where available) via the uniprot_mapping property of the Structure class:
https://github.com/pinder-org/pinder/blob/20487ac403516f453d8be1b0bdc07a9f7a77d9df/src/pinder-core/pinder/core/loader/structure.py#L328-L337
See tutorials covering this here:
https://pinder-org.github.io/pinder/pinder-system.html#structures-have-resolved-pdb2uniprot-and-resolved-uniprot2pdb-properties
https://pinder-org.github.io/pinder/source/pinder.core.index.html#pinder.core.index.utils.IndexEntry.mapping_paths

Q2. How are residues numbered in Pinder PDB files?
Are they aligned with the SEQRES sequence (including missing residues)?
For example, can I assume that residue number 4 in a Pinder PDB corresponds to the fourth residue in the SEQRES sequence?

The residues in pinder PDBs are numbered sequentially.

Are they aligned with the SEQRES sequence (including missing residues)?
No, but mapping and alignment is possible with above.

I believe my answer above should cover the remaining question in Q2, but let me know if you need any further clarification!

Thank you so much for your incredibly kind and detailed response @danielkovtun ! I now have a complete understanding of the concept.
However, I recently realized that, in addition to the methods you mentioned, I can also obtain SEQRES information through the parquet files found in the mappings directory (e.g., mappings/4xl5__A1_P42212-L.parquet). These files seem to contain the information I need, as they also include sequences where resolved = False.
For example, in mappings/4xl5__A1_P42212-L.parquet, I found the following entries:
idx,entry_id,entity_id,asym_id,pdb_strand_id,resi,resi_pdb,resi_auth,resn,one_letter_code_can,resolved,one_letter_code_uniprot,resi_uniprot,uniprot_acc,chain

85,4xl5,1,A,A,86,66,66.0,CRO,T,0,,,,A1
86,4xl5,1,A,A,87,68,68.0,VAL,Y,1,V,68,P42212,A1
87,4xl5,1,A,A,88,69,69.0,GLN,G,1,Q,69,P42212,A1
88,4xl5,1,A,A,89,70,70.0,CYS,V,1,C,70,P42212,A1
89,4xl5,1,A,A,90,71,71.0,PHE,Q,1,F,71,P42212,A1
90,4xl5,1,A,A,91,72,72.0,SER,C,1,S,72,P42212,A1
91,4xl5,1,A,A,92,73,73.0,ARG,F,1,R,73,P42212,A1
92,4xl5,1,A,A,93,74,74.0,TYR,S,1,Y,74,P42212,A1

While reviewing the above data, I noticed that in some cases, the one_letter_code_can differs from the PDB residue.
For example, for index 86, resi_pdb = VAL, but one_letter_code_can = Y.
I'm curious why this discrepancy exists. Is this a simple error, or is there a specific reason behind it?
Thank you once again for your time and thoughtful response!

No problem, glad it was helpful!
Regarding the example above, this seems to be an odd case where there are modified / mutated residues that deviate from the canonical uniprot sequence.
If you look at the mapping table, for resi [84, 85, 86, 87, 88], you can see there is a disconnect where there is no equivalent uniprot residue:
>>> print(mapping.query('resi in [84, 85, 86, 87, 88]').to_markdown(index=False))

entry_id
entity_id
asym_id
pdb_strand_id
resi
resi_pdb
resi_auth
resn
one_letter_code_can
resolved
one_letter_code_uniprot
resi_uniprot
uniprot_acc
chain

4xl5
1
A
A
84
63
63
THR
T
1
T
63
P42212
A1

4xl5
1
A
A
85
64
64
LEU
L
1
F
64
P42212
A1

4xl5
1
A
A
86
66
66
CRO
T
0
nan

nan
A1

4xl5
1
A
A
87
68
68
VAL
Y
1
V
68
P42212
A1

4xl5
1
A
A
88
69
69
GLN
G
1
Q
69
P42212
A1

If we compare the PDB structure provided in pinder with the original RCSB structure, we see that residue 65 and 67 is missing, and the codes go LEU.64, CRO.66, VAL.68.

I believe in this case its a bug in the mapping parquet because the CRO residue is a covalently bonded modified residue which spans the canonical uniprot sequence between residue 65-67 and while resi and resi_pdb do align with the structure in pinder and RCSB, the three missing codes are "mis-aligned" with the numbering of the PDB file (T, Y, G are not in the structure, but represented by CRO).
If you needed the original sequence here, I believe the best source would be the supplemental entity metadata I mentioned above:
>>> entity_meta.query('entry_id == "4xl5" and asym_id == "A"').sequence.values[0]
'MRGSHHHHHHTDPHASSAGSPVSKGEELFTGVVPILVELDGDVNGHKFSVSGEGEGDATYGKLTLKFICTTGKLPVPWPTLVTTLTYGVQCFSRYPDHMKQHDFFKSAMPEGYVQERTIFFKDDGNYKTRAEVKFEGDTLVNRIELKGIDFKEDGNILGHKLEYNYNSHNVYIMADKQKNGIKVNFKIRHNIEDGSVQLADHYQQNTPIGDGPVLLPDNHYLSTQSALSKDPNEKRDHMVLLEFVTAAGITLGMDELYKGSGASEQKLISEEDL'

Thanks for flagging this, it will be worth looking into the original next-gen mmCIF file for this entry to come up with a more robust extraction for a future release.

I see, that makes perfect sense now. Thank you so much for your explanation, it was very helpful!!